2024-05-04T17:29:20.311 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-04T17:29:20.311 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} --name yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124 -- /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124/orig.config.yaml 2024-05-04T17:29:20.351 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-04T17:29:20.442 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124 branch: wip-yuriw11-testing-20240501.200505-squid description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7687124' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuriw11-testing-20240501.200505-squid ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: f273489c6dcd4bc88409993babd09dd99491162c valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: debuginfo: true flavor: default sha1: f273489c6dcd4bc88409993babd09dd99491162c selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuriw11-testing-20240501.200505-squid sha1: f273489c6dcd4bc88409993babd09dd99491162c owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8797 sha1: f273489c6dcd4bc88409993babd09dd99491162c sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuriw11-testing-20240501.200505-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f273489c6dcd4bc88409993babd09dd99491162c targets: smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOvty0GMgfRqiueBK+9zrw9w+jUpACO7qhUu4rXcOfp2gxDXptEzxgA9fTtDwyweBBm7WSF980wwDx3iJ6RO9Bw= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHYzvai/rL2inX2fbEzwz6S4DYA4Fbh2e/rKxOZaVRoBX4sR4J8ehcnFxUBq0G36PrKUFqP287tjivJTsIZ7S2w= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-02_23:59:28 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-04T17:29:20.443 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa; will attempt to use it 2024-05-04T17:29:20.443 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa/tasks 2024-05-04T17:29:20.443 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-04T17:29:20.445 INFO:teuthology.task.internal:Checking packages... 2024-05-04T17:29:20.470 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f273489c6dcd4bc88409993babd09dd99491162c' 2024-05-04T17:29:20.471 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-04T17:29:20.471 INFO:teuthology.packaging:ref: None 2024-05-04T17:29:20.471 INFO:teuthology.packaging:tag: None 2024-05-04T17:29:20.471 INFO:teuthology.packaging:branch: wip-yuriw11-testing-20240501.200505-squid 2024-05-04T17:29:20.471 INFO:teuthology.packaging:sha1: f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:29:20.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuriw11-testing-20240501.200505-squid 2024-05-04T17:29:20.620 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2497.gf273489c 2024-05-04T17:29:20.622 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-04T17:29:20.631 INFO:teuthology.task.internal:no buildpackages task found 2024-05-04T17:29:20.631 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-04T17:29:20.666 INFO:teuthology.task.internal:Saving configuration 2024-05-04T17:29:20.684 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-04T17:29:20.703 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-04T17:29:20.728 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-04 17:24:11.240129', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOvty0GMgfRqiueBK+9zrw9w+jUpACO7qhUu4rXcOfp2gxDXptEzxgA9fTtDwyweBBm7WSF980wwDx3iJ6RO9Bw='} 2024-05-04T17:29:20.749 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-04 17:24:11.238694', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHYzvai/rL2inX2fbEzwz6S4DYA4Fbh2e/rKxOZaVRoBX4sR4J8ehcnFxUBq0G36PrKUFqP287tjivJTsIZ7S2w='} 2024-05-04T17:29:20.749 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-04T17:29:20.761 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-05-04T17:29:20.761 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-05-04T17:29:20.761 INFO:teuthology.run_tasks:Running task console_log... 2024-05-04T17:29:20.859 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f931f1d1f70>, signals=[15]) 2024-05-04T17:29:20.860 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-04T17:29:20.872 INFO:teuthology.task.internal:Opening connections... 2024-05-04T17:29:20.872 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2024-05-04T17:29:20.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T17:29:20.961 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-05-04T17:29:20.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T17:29:21.036 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-04T17:29:21.048 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2024-05-04T17:29:21.067 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2024-05-04T17:29:21.067 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2024-05-04T17:29:21.125 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream" 2024-05-04T17:29:21.126 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="9" 2024-05-04T17:29:21.126 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2024-05-04T17:29:21.126 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2024-05-04T17:29:21.126 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="9" 2024-05-04T17:29:21.127 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el9" 2024-05-04T17:29:21.127 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-04T17:29:21.127 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2024-05-04T17:29:21.127 INFO:teuthology.orchestra.run.smithi019.stdout:LOGO="fedora-logo-icon" 2024-05-04T17:29:21.127 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-04T17:29:21.127 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2024-05-04T17:29:21.128 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-04T17:29:21.128 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-04T17:29:21.128 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-04T17:29:21.128 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2024-05-04T17:29:21.164 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-05-04T17:29:21.181 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-05-04T17:29:21.182 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-05-04T17:29:21.238 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-05-04T17:29:21.238 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-05-04T17:29:21.238 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-05-04T17:29:21.239 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-05-04T17:29:21.239 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-05-04T17:29:21.239 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-05-04T17:29:21.239 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-04T17:29:21.239 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-05-04T17:29:21.239 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-05-04T17:29:21.239 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-04T17:29:21.239 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-05-04T17:29:21.239 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-04T17:29:21.239 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-04T17:29:21.239 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-04T17:29:21.239 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-05-04T17:29:21.263 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-04T17:29:21.281 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-04T17:29:21.310 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-04T17:29:21.310 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2024-05-04T17:29:21.313 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-05-04T17:29:21.330 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-04T17:29:21.344 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-04T17:29:21.344 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2024-05-04T17:29:21.371 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-05-04T17:29:21.396 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-04T17:29:21.447 INFO:teuthology.run_tasks:Running task kernel... 2024-05-04T17:29:21.467 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-04T17:29:21.468 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-04T17:29:21.468 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-04T17:29:21.468 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-04T17:29:21.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:29:21.485 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-05-04T17:29:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:29:21.487 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-05-04T17:29:21.542 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-437.el9.x86_64 2024-05-04T17:29:21.545 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.14.0-437.el9.x86_64 2024-05-04T17:29:21.545 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum install -y kernel 2024-05-04T17:29:21.550 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-437.el9.x86_64 2024-05-04T17:29:21.550 INFO:teuthology.task.kernel:Running kernel on smithi019: 5.14.0-437.el9.x86_64 2024-05-04T17:29:21.550 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum install -y kernel 2024-05-04T17:29:24.099 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 8.0 MB 00:00 2024-05-04T17:29:25.110 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-05-04T17:29:26.125 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-05-04T17:29:30.309 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-05-04T17:29:31.835 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-05-04T17:29:32.169 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 3.2 MB/s | 19 MB 00:05 2024-05-04T17:29:33.353 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-05-04T17:29:36.353 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.2 MB 00:00 2024-05-04T17:29:37.895 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-05-04T17:29:38.603 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-05-04T17:29:40.438 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 8.7 MB/s | 21 MB 00:02 2024-05-04T17:29:40.543 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-04T17:29:40.543 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-04T17:29:40.645 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:29:40.646 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-05-04T17:29:40.646 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:29:40.780 DEBUG:teuthology.orchestra.run.smithi019:> echo no | sudo yum reinstall kernel || true 2024-05-04T17:29:41.301 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:03 ago on Sat 04 May 2024 05:29:38 PM UTC. 2024-05-04T17:29:41.432 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:29:41.440 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:29:41.441 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-05-04T17:29:41.441 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:29:41.441 INFO:teuthology.orchestra.run.smithi019.stdout:Reinstalling: 2024-05-04T17:29:41.441 INFO:teuthology.orchestra.run.smithi019.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-04T17:29:41.441 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:29:41.441 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:29:41.441 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:29:41.441 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:29:41.441 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 255 k 2024-05-04T17:29:41.441 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 0 2024-05-04T17:29:41.441 INFO:teuthology.orchestra.run.smithi019.stderr:Operation aborted. 2024-05-04T17:29:41.499 INFO:teuthology.orchestra.run.smithi019.stdout:Is this ok [y/N]: 2024-05-04T17:29:41.499 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum reinstall -y kernel || true 2024-05-04T17:29:42.012 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:04 ago on Sat 04 May 2024 05:29:38 PM UTC. 2024-05-04T17:29:42.132 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:29:42.133 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:29:42.133 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-05-04T17:29:42.133 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:29:42.133 INFO:teuthology.orchestra.run.smithi019.stdout:Reinstalling: 2024-05-04T17:29:42.133 INFO:teuthology.orchestra.run.smithi019.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-04T17:29:42.133 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:29:42.134 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:29:42.134 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:29:42.134 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:29:42.134 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 255 k 2024-05-04T17:29:42.134 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 0 2024-05-04T17:29:42.134 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:29:42.580 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.4 MB/s | 255 kB 00:00 2024-05-04T17:29:42.580 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:29:42.580 INFO:teuthology.orchestra.run.smithi019.stdout:Total 571 kB/s | 255 kB 00:00 2024-05-04T17:29:42.600 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:29:42.782 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:29:42.782 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:29:42.787 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:29:42.788 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:29:42.936 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:29:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-04T17:29:43.022 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T17:29:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T17:29:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-04T17:29:44.426 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T17:29:44.427 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:29:44.427 INFO:teuthology.orchestra.run.smithi019.stdout:Reinstalled: 2024-05-04T17:29:44.427 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-04T17:29:44.427 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:29:44.427 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:29:44.613 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-05-04T17:29:44.661 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-04T17:29:44.662 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-04T17:29:44.662 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-04T17:29:44.662 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-04T17:29:44.662 DEBUG:teuthology.orchestra.run.smithi019:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-04T17:29:44.692 INFO:teuthology.orchestra.run.smithi019.stdout:ttyS1 2024-05-04T17:29:44.713 DEBUG:teuthology.parallel:result is None 2024-05-04T17:29:45.555 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-05-04T17:29:47.427 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-04T17:29:47.427 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-04T17:29:47.520 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:29:47.521 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-05-04T17:29:47.521 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:29:47.630 DEBUG:teuthology.orchestra.run.smithi186:> echo no | sudo yum reinstall kernel || true 2024-05-04T17:29:48.131 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:03 ago on Sat 04 May 2024 05:29:45 PM UTC. 2024-05-04T17:29:48.250 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:29:48.263 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:29:48.263 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-05-04T17:29:48.263 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:29:48.263 INFO:teuthology.orchestra.run.smithi186.stdout:Reinstalling: 2024-05-04T17:29:48.263 INFO:teuthology.orchestra.run.smithi186.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-04T17:29:48.263 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:29:48.263 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:29:48.263 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:29:48.264 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:29:48.264 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 255 k 2024-05-04T17:29:48.264 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 0 2024-05-04T17:29:48.264 INFO:teuthology.orchestra.run.smithi186.stderr:Operation aborted. 2024-05-04T17:29:48.323 INFO:teuthology.orchestra.run.smithi186.stdout:Is this ok [y/N]: 2024-05-04T17:29:48.323 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum reinstall -y kernel || true 2024-05-04T17:29:48.815 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:03 ago on Sat 04 May 2024 05:29:45 PM UTC. 2024-05-04T17:29:48.930 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:29:48.931 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:29:48.931 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-05-04T17:29:48.931 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:29:48.931 INFO:teuthology.orchestra.run.smithi186.stdout:Reinstalling: 2024-05-04T17:29:48.932 INFO:teuthology.orchestra.run.smithi186.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-04T17:29:48.932 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:29:48.932 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:29:48.932 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:29:48.932 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:29:48.932 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 255 k 2024-05-04T17:29:48.932 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 0 2024-05-04T17:29:48.932 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:29:49.223 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.1 MB/s | 255 kB 00:00 2024-05-04T17:29:49.224 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:29:49.224 INFO:teuthology.orchestra.run.smithi186.stdout:Total 872 kB/s | 255 kB 00:00 2024-05-04T17:29:49.241 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:29:49.447 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:29:49.448 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:29:49.452 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:29:49.453 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:29:49.582 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:29:49.682 INFO:teuthology.orchestra.run.smithi186.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-04T17:29:49.683 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T17:29:50.426 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T17:29:50.426 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-04T17:29:51.063 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-04T17:29:51.064 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:29:51.064 INFO:teuthology.orchestra.run.smithi186.stdout:Reinstalled: 2024-05-04T17:29:51.064 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-04T17:29:51.064 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:29:51.064 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:29:51.227 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-05-04T17:29:51.315 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-04T17:29:51.315 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-04T17:29:51.315 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-04T17:29:51.316 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-04T17:29:51.316 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-04T17:29:51.357 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2024-05-04T17:29:51.387 DEBUG:teuthology.parallel:result is None 2024-05-04T17:29:51.387 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-04T17:29:51.395 INFO:teuthology.task.internal:Creating test directory... 2024-05-04T17:29:51.395 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-04T17:29:51.398 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-04T17:29:51.441 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-04T17:29:51.449 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-04T17:29:51.520 INFO:teuthology.task.internal:Creating archive directory... 2024-05-04T17:29:51.520 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-04T17:29:51.523 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-04T17:29:51.549 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-04T17:29:51.560 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-04T17:29:51.560 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-04T17:29:51.587 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-04T17:29:51.617 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-04T17:29:51.622 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-04T17:29:51.626 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-04T17:29:51.634 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-04T17:29:51.635 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-04T17:29:51.642 INFO:teuthology.task.internal:Configuring sudo... 2024-05-04T17:29:51.643 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-04T17:29:51.678 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-04T17:29:51.704 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-04T17:29:51.713 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-04T17:29:51.714 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-04T17:29:51.747 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-04T17:29:51.763 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T17:29:51.846 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T17:29:51.894 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T17:29:51.976 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T17:29:52.019 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:29:52.019 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-04T17:29:52.085 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T17:29:52.123 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-04T17:29:52.176 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T17:29:52.255 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-04T17:29:52.297 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:29:52.297 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-04T17:29:52.360 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2024-05-04T17:29:52.363 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-05-04T17:29:52.403 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-04T17:29:52.436 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-04T17:29:52.890 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-04T17:29:52.898 INFO:teuthology.task.internal:Starting timer... 2024-05-04T17:29:52.898 INFO:teuthology.run_tasks:Running task pcp... 2024-05-04T17:29:52.941 INFO:teuthology.run_tasks:Running task selinux... 2024-05-04T17:29:52.990 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-05-04T17:29:52.990 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2024-05-04T17:29:53.040 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2024-05-04T17:29:53.042 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-05-04T17:29:53.114 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-05-04T17:29:53.115 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-04T17:29:53.115 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2024-05-04T17:29:53.154 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2024-05-04T17:29:53.154 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-05-04T17:29:53.174 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-05-04T17:29:53.175 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi019.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-05-04T17:29:53.175 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-04T17:29:53.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:29:53.219 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-04T17:29:53.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:29:53.246 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-04T17:29:53.246 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2024-05-04T17:29:53.285 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-05-04T17:29:53.311 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-04T17:29:53.320 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-04T17:29:53.338 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-04T17:29:53.340 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi019.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-04T17:33:46.970 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi019.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-05-04T17:33:46.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-05-04T17:33:46.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T17:33:47.054 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-05-04T17:33:47.137 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-05-04T17:33:47.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-04T17:33:47.138 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-04T17:33:47.215 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-05-04T17:33:47.297 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-04T17:33:47.298 INFO:teuthology.run_tasks:Running task clock... 2024-05-04T17:33:47.309 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-04T17:33:47.309 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-04T17:33:47.309 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-04T17:33:47.312 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-04T17:33:47.312 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-04T17:33:47.338 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-04T17:33:47.352 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-04T17:33:47.365 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-04T17:33:47.375 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2024-05-04T17:33:47.378 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-04T17:33:47.386 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2024-05-04T17:33:47.399 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-04T17:33:47.401 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-05-04T17:33:47.412 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-05-04T17:33:47.412 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-04T17:33:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-04T17:33:47.438 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-04T17:33:47.473 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-05-04T17:33:47.477 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-04T17:33:47.477 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-05-04T17:33:47.477 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:47.477 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:47.478 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:47.478 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:47.493 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-05-04T17:33:47.496 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-04T17:33:47.496 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-05-04T17:33:47.496 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:47.496 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:47.496 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:47.496 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T17:33:47.497 INFO:teuthology.run_tasks:Running task install... 2024-05-04T17:33:47.506 DEBUG:teuthology.task.install:project ceph 2024-05-04T17:33:47.507 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c'}} 2024-05-04T17:33:47.507 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c'} 2024-05-04T17:33:47.507 INFO:teuthology.task.install:Using flavor: default 2024-05-04T17:33:47.512 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-05-04T17:33:47.513 INFO:teuthology.task.install:extra packages: [] 2024-05-04T17:33:47.513 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c', 'tag': None, 'wait_for_package': False} 2024-05-04T17:33:47.514 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:33:47.516 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c', 'tag': None, 'wait_for_package': False} 2024-05-04T17:33:47.516 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:33:47.661 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/ 2024-05-04T17:33:47.661 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2497.gf273489c 2024-05-04T17:33:47.666 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/ 2024-05-04T17:33:47.666 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2497.gf273489c 2024-05-04T17:33:47.902 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-04T17:33:47.903 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:33:47.903 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-04T17:33:47.928 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-05-04T17:33:47.928 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f273489c6dcd4bc88409993babd09dd99491162c/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-04T17:33:47.982 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuriw11-testing-20240501.200505-squid/f273489c6dcd4bc88409993babd09dd99491162c/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-04T17:33:47.982 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:33:47.982 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-04T17:33:47.994 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-04T17:33:48.009 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-05-04T17:33:48.009 DEBUG:teuthology.orchestra.run.smithi019:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f273489c6dcd4bc88409993babd09dd99491162c/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-04T17:33:48.068 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-04T17:33:48.075 DEBUG:teuthology.orchestra.run.smithi019:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-04T17:33:48.130 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-05-04T17:33:48.131 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-05-04T17:33:48.150 DEBUG:teuthology.orchestra.run.smithi019:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-04T17:33:48.214 INFO:teuthology.orchestra.run.smithi019.stdout:check_obsoletes = 1 2024-05-04T17:33:48.215 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum clean all 2024-05-04T17:33:48.360 INFO:teuthology.orchestra.run.smithi186.stdout:50 files removed 2024-05-04T17:33:48.388 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw 2024-05-04T17:33:48.451 INFO:teuthology.orchestra.run.smithi019.stdout:50 files removed 2024-05-04T17:33:48.478 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw 2024-05-04T17:33:49.458 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 103 kB/s | 83 kB 00:00 2024-05-04T17:33:49.810 INFO:teuthology.orchestra.run.smithi019.stdout:ceph packages for x86_64 79 kB/s | 83 kB 00:01 2024-05-04T17:33:50.200 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 17 kB/s | 11 kB 00:00 2024-05-04T17:33:50.661 INFO:teuthology.orchestra.run.smithi019.stdout:ceph noarch packages 14 kB/s | 11 kB 00:00 2024-05-04T17:33:51.001 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-05-04T17:33:51.262 INFO:teuthology.orchestra.run.smithi019.stdout:ceph source packages 3.4 kB/s | 1.9 kB 00:00 2024-05-04T17:33:52.039 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-05-04T17:33:52.734 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-05-04T17:33:54.371 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-05-04T17:33:55.110 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-05-04T17:33:58.797 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.2 MB 00:00 2024-05-04T17:33:59.337 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.2 MB 00:00 2024-05-04T17:34:00.381 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-05-04T17:34:01.138 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-05-04T17:34:01.502 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-05-04T17:34:02.924 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-05-04T17:34:06.741 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-05-04T17:34:08.030 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-05-04T17:34:08.995 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:34:08.998 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:34:08.998 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-05-04T17:34:08.998 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:34:08.998 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:34:08.998 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw x86_64 2:19.0.0-2497.gf273489c.el9 ceph 10 M 2024-05-04T17:34:08.998 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2024-05-04T17:34:08.998 INFO:teuthology.orchestra.run.smithi019.stdout: librados2 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 3.4 M 2024-05-04T17:34:08.998 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 3.1 M 2024-05-04T17:34:08.998 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base x86_64 2:19.0.0-2497.gf273489c.el9 ceph 5.4 M 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common x86_64 2:19.0.0-2497.gf273489c.el9 ceph 21 M 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux x86_64 2:19.0.0-2497.gf273489c.el9 ceph 25 k 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 745 k 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 502 k 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-04T17:34:08.999 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 5.3 M 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse x86_64 2:19.0.0-2497.gf273489c.el9 ceph 45 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common x86_64 2:19.0.0-2497.gf273489c.el9 ceph 135 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs x86_64 2:19.0.0-2497.gf273489c.el9 ceph 162 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados x86_64 2:19.0.0-2497.gf273489c.el9 ceph 321 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd x86_64 2:19.0.0-2497.gf273489c.el9 ceph 298 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw x86_64 2:19.0.0-2497.gf273489c.el9 ceph 99 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-04T17:34:09.000 INFO:teuthology.orchestra.run.smithi019.stdout:Installing weak dependencies: 2024-05-04T17:34:09.001 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-04T17:34:09.001 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:09.001 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:34:09.001 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:34:09.001 INFO:teuthology.orchestra.run.smithi019.stdout:Install 30 Packages 2024-05-04T17:34:09.001 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 2 Packages 2024-05-04T17:34:09.001 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:09.001 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 60 M 2024-05-04T17:34:09.001 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:34:10.189 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:34:10.191 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:34:10.191 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-04T17:34:10.191 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:34:10.191 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-2497.gf273489c.el9 ceph 10 M 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 3.4 M 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 3.1 M 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-2497.gf273489c.el9 ceph 5.4 M 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-2497.gf273489c.el9 ceph 21 M 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-2497.gf273489c.el9 ceph 25 k 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 745 k 2024-05-04T17:34:10.192 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 502 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-2497.gf273489c.el9 ceph 5.3 M 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:19.0.0-2497.gf273489c.el9 ceph 45 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-2497.gf273489c.el9 ceph 135 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-2497.gf273489c.el9 ceph 162 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-2497.gf273489c.el9 ceph 321 k 2024-05-04T17:34:10.193 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-2497.gf273489c.el9 ceph 298 k 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-2497.gf273489c.el9 ceph 99 k 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout:Install 30 Packages 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 2 Packages 2024-05-04T17:34:10.194 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:10.195 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 60 M 2024-05-04T17:34:10.195 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:34:11.951 INFO:teuthology.orchestra.run.smithi019.stdout:(1/32): ceph-base-19.0.0-2497.gf273489c.el9.x86 2.1 MB/s | 5.4 MB 00:02 2024-05-04T17:34:12.034 INFO:teuthology.orchestra.run.smithi019.stdout:(2/32): ceph-selinux-19.0.0-2497.gf273489c.el9. 302 kB/s | 25 kB 00:00 2024-05-04T17:34:12.314 INFO:teuthology.orchestra.run.smithi186.stdout:(1/32): ceph-base-19.0.0-2497.gf273489c.el9.x86 2.6 MB/s | 5.4 MB 00:02 2024-05-04T17:34:12.351 INFO:teuthology.orchestra.run.smithi019.stdout:(3/32): libcephfs2-19.0.0-2497.gf273489c.el9.x8 2.3 MB/s | 745 kB 00:00 2024-05-04T17:34:12.381 INFO:teuthology.orchestra.run.smithi186.stdout:(2/32): ceph-selinux-19.0.0-2497.gf273489c.el9. 378 kB/s | 25 kB 00:00 2024-05-04T17:34:12.609 INFO:teuthology.orchestra.run.smithi019.stdout:(4/32): libradosstriper1-19.0.0-2497.gf273489c. 1.9 MB/s | 502 kB 00:00 2024-05-04T17:34:12.615 INFO:teuthology.orchestra.run.smithi186.stdout:(3/32): libcephfs2-19.0.0-2497.gf273489c.el9.x8 3.1 MB/s | 745 kB 00:00 2024-05-04T17:34:12.815 INFO:teuthology.orchestra.run.smithi186.stdout:(4/32): libradosstriper1-19.0.0-2497.gf273489c. 2.4 MB/s | 502 kB 00:00 2024-05-04T17:34:13.577 INFO:teuthology.orchestra.run.smithi019.stdout:(5/32): ceph-common-19.0.0-2497.gf273489c.el9.x 5.0 MB/s | 21 MB 00:04 2024-05-04T17:34:13.643 INFO:teuthology.orchestra.run.smithi019.stdout:(6/32): python3-ceph-argparse-19.0.0-2497.gf273 680 kB/s | 45 kB 00:00 2024-05-04T17:34:13.702 INFO:teuthology.orchestra.run.smithi019.stdout:(7/32): python3-ceph-common-19.0.0-2497.gf27348 2.3 MB/s | 135 kB 00:00 2024-05-04T17:34:13.760 INFO:teuthology.orchestra.run.smithi019.stdout:(8/32): python3-cephfs-19.0.0-2497.gf273489c.el 2.7 MB/s | 162 kB 00:00 2024-05-04T17:34:13.836 INFO:teuthology.orchestra.run.smithi019.stdout:(9/32): python3-rados-19.0.0-2497.gf273489c.el9 4.2 MB/s | 321 kB 00:00 2024-05-04T17:34:13.911 INFO:teuthology.orchestra.run.smithi019.stdout:(10/32): python3-rbd-19.0.0-2497.gf273489c.el9. 3.9 MB/s | 298 kB 00:00 2024-05-04T17:34:13.970 INFO:teuthology.orchestra.run.smithi019.stdout:(11/32): python3-rgw-19.0.0-2497.gf273489c.el9. 1.7 MB/s | 99 kB 00:00 2024-05-04T17:34:14.078 INFO:teuthology.orchestra.run.smithi019.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 988 kB/s | 107 kB 00:00 2024-05-04T17:34:14.129 INFO:teuthology.orchestra.run.smithi019.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-05-04T17:34:14.170 INFO:teuthology.orchestra.run.smithi019.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-05-04T17:34:14.232 INFO:teuthology.orchestra.run.smithi186.stdout:(5/32): librgw2-19.0.0-2497.gf273489c.el9.x86_6 3.8 MB/s | 5.3 MB 00:01 2024-05-04T17:34:14.238 INFO:teuthology.orchestra.run.smithi019.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-05-04T17:34:14.279 INFO:teuthology.orchestra.run.smithi019.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.9 MB/s | 292 kB 00:00 2024-05-04T17:34:14.282 INFO:teuthology.orchestra.run.smithi186.stdout:(6/32): python3-ceph-argparse-19.0.0-2497.gf273 902 kB/s | 45 kB 00:00 2024-05-04T17:34:14.321 INFO:teuthology.orchestra.run.smithi019.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-05-04T17:34:14.349 INFO:teuthology.orchestra.run.smithi186.stdout:(7/32): python3-ceph-common-19.0.0-2497.gf27348 2.0 MB/s | 135 kB 00:00 2024-05-04T17:34:14.363 INFO:teuthology.orchestra.run.smithi019.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-05-04T17:34:14.405 INFO:teuthology.orchestra.run.smithi019.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 861 kB/s | 36 kB 00:00 2024-05-04T17:34:14.408 INFO:teuthology.orchestra.run.smithi186.stdout:(8/32): python3-cephfs-19.0.0-2497.gf273489c.el 2.7 MB/s | 162 kB 00:00 2024-05-04T17:34:14.447 INFO:teuthology.orchestra.run.smithi019.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-05-04T17:34:14.508 INFO:teuthology.orchestra.run.smithi186.stdout:(9/32): python3-rados-19.0.0-2497.gf273489c.el9 3.1 MB/s | 321 kB 00:00 2024-05-04T17:34:14.598 INFO:teuthology.orchestra.run.smithi019.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.0 MB/s | 309 kB 00:00 2024-05-04T17:34:14.608 INFO:teuthology.orchestra.run.smithi186.stdout:(10/32): python3-rbd-19.0.0-2497.gf273489c.el9. 2.9 MB/s | 298 kB 00:00 2024-05-04T17:34:14.667 INFO:teuthology.orchestra.run.smithi186.stdout:(11/32): python3-rgw-19.0.0-2497.gf273489c.el9. 1.7 MB/s | 99 kB 00:00 2024-05-04T17:34:14.740 INFO:teuthology.orchestra.run.smithi019.stdout:(22/32): librgw2-19.0.0-2497.gf273489c.el9.x86_ 2.5 MB/s | 5.3 MB 00:02 2024-05-04T17:34:14.776 INFO:teuthology.orchestra.run.smithi186.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 988 kB/s | 107 kB 00:00 2024-05-04T17:34:14.815 INFO:teuthology.orchestra.run.smithi019.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-05-04T17:34:14.834 INFO:teuthology.orchestra.run.smithi186.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-05-04T17:34:14.868 INFO:teuthology.orchestra.run.smithi186.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-05-04T17:34:14.899 INFO:teuthology.orchestra.run.smithi019.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 583 kB/s | 49 kB 00:00 2024-05-04T17:34:14.943 INFO:teuthology.orchestra.run.smithi186.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-05-04T17:34:14.957 INFO:teuthology.orchestra.run.smithi019.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-05-04T17:34:14.985 INFO:teuthology.orchestra.run.smithi186.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.9 MB/s | 292 kB 00:00 2024-05-04T17:34:15.027 INFO:teuthology.orchestra.run.smithi186.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-05-04T17:34:15.069 INFO:teuthology.orchestra.run.smithi186.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-05-04T17:34:15.108 INFO:teuthology.orchestra.run.smithi019.stdout:(26/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.5 MB/s | 839 kB 00:00 2024-05-04T17:34:15.111 INFO:teuthology.orchestra.run.smithi186.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 866 kB/s | 36 kB 00:00 2024-05-04T17:34:15.153 INFO:teuthology.orchestra.run.smithi186.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-05-04T17:34:15.166 INFO:teuthology.orchestra.run.smithi019.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-05-04T17:34:15.366 INFO:teuthology.orchestra.run.smithi019.stdout:(28/32): libarrow-9.0.0-11.el9.x86_64.rpm 5.7 MB/s | 4.4 MB 00:00 2024-05-04T17:34:15.475 INFO:teuthology.orchestra.run.smithi019.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 5.1 MB/s | 1.6 MB 00:00 2024-05-04T17:34:15.495 INFO:teuthology.orchestra.run.smithi186.stdout:(21/32): ceph-radosgw-19.0.0-2497.gf273489c.el9 2.0 MB/s | 10 MB 00:05 2024-05-04T17:34:15.520 INFO:teuthology.orchestra.run.smithi186.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 842 kB/s | 309 kB 00:00 2024-05-04T17:34:15.579 INFO:teuthology.orchestra.run.smithi186.stdout:(23/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-05-04T17:34:15.675 INFO:teuthology.orchestra.run.smithi186.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 506 kB/s | 49 kB 00:00 2024-05-04T17:34:15.737 INFO:teuthology.orchestra.run.smithi186.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-05-04T17:34:15.784 INFO:teuthology.orchestra.run.smithi019.stdout:(30/32): librbd1-19.0.0-2497.gf273489c.el9.x86_ 10 MB/s | 3.1 MB 00:00 2024-05-04T17:34:15.946 INFO:teuthology.orchestra.run.smithi186.stdout:(26/32): ceph-common-19.0.0-2497.gf273489c.el9. 3.7 MB/s | 21 MB 00:05 2024-05-04T17:34:15.980 INFO:teuthology.orchestra.run.smithi186.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.4 MB/s | 839 kB 00:00 2024-05-04T17:34:16.105 INFO:teuthology.orchestra.run.smithi186.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-05-04T17:34:16.164 INFO:teuthology.orchestra.run.smithi186.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 883 kB/s | 192 kB 00:00 2024-05-04T17:34:16.239 INFO:teuthology.orchestra.run.smithi186.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2024-05-04T17:34:16.334 INFO:teuthology.orchestra.run.smithi019.stdout:(31/32): ceph-radosgw-19.0.0-2497.gf273489c.el9 1.5 MB/s | 10 MB 00:06 2024-05-04T17:34:16.384 INFO:teuthology.orchestra.run.smithi019.stdout:(32/32): librados2-19.0.0-2497.gf273489c.el9.x8 3.3 MB/s | 3.4 MB 00:01 2024-05-04T17:34:16.387 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:34:16.388 INFO:teuthology.orchestra.run.smithi019.stdout:Total 8.1 MB/s | 60 MB 00:07 2024-05-04T17:34:16.495 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:34:16.536 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:34:16.536 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:34:16.556 INFO:teuthology.orchestra.run.smithi186.stdout:(31/32): librbd1-19.0.0-2497.gf273489c.el9.x86_ 8.0 MB/s | 3.1 MB 00:00 2024-05-04T17:34:16.664 INFO:teuthology.orchestra.run.smithi186.stdout:(32/32): librados2-19.0.0-2497.gf273489c.el9.x8 6.1 MB/s | 3.4 MB 00:00 2024-05-04T17:34:16.668 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:34:16.668 INFO:teuthology.orchestra.run.smithi186.stdout:Total 9.2 MB/s | 60 MB 00:06 2024-05-04T17:34:16.773 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:34:16.810 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:34:16.811 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:34:17.025 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:34:17.026 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:34:17.272 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:34:17.273 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:34:18.557 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:34:18.636 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-04T17:34:18.673 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-04T17:34:18.715 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-04T17:34:18.737 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-04T17:34:18.761 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:34:18.764 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-04T17:34:18.832 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-05-04T17:34:18.869 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-05-04T17:34:18.895 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-05-04T17:34:18.918 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-04T17:34:18.942 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-05-04T17:34:18.964 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-04T17:34:19.018 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-04T17:34:19.134 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-04T17:34:19.197 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-05-04T17:34:19.415 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-04T17:34:19.491 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librados2-2:19.0.0-2497.gf273489c.el9.x86_64 7/34 2024-05-04T17:34:19.584 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:19.0.0-2497.gf273489c.el9.x86_64 7/34 2024-05-04T17:34:19.588 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-05-04T17:34:19.638 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-2497.gf273489c.el9.x86_64 7/34 2024-05-04T17:34:19.673 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 8/34 2024-05-04T17:34:19.701 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 9/34 2024-05-04T17:34:19.742 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-2497.gf273489c.el9.x86_64 7/34 2024-05-04T17:34:19.755 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 9/34 2024-05-04T17:34:19.792 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-04T17:34:19.837 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-argparse-2:19.0.0-2497.gf273489c.el9. 11/34 2024-05-04T17:34:19.840 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 8/34 2024-05-04T17:34:19.873 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 9/34 2024-05-04T17:34:19.906 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_64 12/34 2024-05-04T17:34:19.928 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86_6 13/34 2024-05-04T17:34:19.936 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 9/34 2024-05-04T17:34:19.973 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-05-04T17:34:19.983 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86_6 13/34 2024-05-04T17:34:20.016 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-2497.gf273489c.el9. 11/34 2024-05-04T17:34:20.018 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-04T17:34:20.085 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_64 12/34 2024-05-04T17:34:20.109 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86_6 13/34 2024-05-04T17:34:20.129 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-04T17:34:20.161 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-04T17:34:20.173 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86_6 13/34 2024-05-04T17:34:20.202 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-04T17:34:20.207 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-05-04T17:34:20.234 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-04T17:34:20.264 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-04T17:34:20.296 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-04T17:34:20.316 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-05-04T17:34:20.341 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-05-04T17:34:20.381 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-05-04T17:34:20.406 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-05-04T17:34:20.436 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-05-04T17:34:20.468 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-05-04T17:34:20.626 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-04T17:34:20.667 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 22/34 2024-05-04T17:34:20.742 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 22/34 2024-05-04T17:34:20.763 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 23/34 2024-05-04T17:34:20.785 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-05-04T17:34:20.808 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-04T17:34:20.831 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 22/34 2024-05-04T17:34:20.922 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 22/34 2024-05-04T17:34:20.953 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 23/34 2024-05-04T17:34:20.989 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-05-04T17:34:21.306 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-04T17:34:21.336 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 26/34 2024-05-04T17:34:21.391 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 26/34 2024-05-04T17:34:21.461 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-05-04T17:34:21.472 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 27/34 2024-05-04T17:34:21.484 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 26/34 2024-05-04T17:34:21.497 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-common-2:19.0.0-2497.gf273489c.el9.x8 28/34 2024-05-04T17:34:21.538 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 26/34 2024-05-04T17:34:21.617 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 27/34 2024-05-04T17:34:21.653 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-2497.gf273489c.el9.x8 28/34 2024-05-04T17:34:23.876 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 29/34 2024-05-04T17:34:23.908 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 29/34 2024-05-04T17:34:23.999 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 29/34 2024-05-04T17:34:24.215 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 29/34 2024-05-04T17:34:24.452 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 29/34 2024-05-04T17:34:24.486 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 30/34 2024-05-04T17:34:24.577 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 30/34 2024-05-04T17:34:24.577 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-04T17:34:24.578 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-04T17:34:24.578 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:24.604 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 31/34 2024-05-04T17:34:24.787 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 29/34 2024-05-04T17:34:24.822 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 30/34 2024-05-04T17:34:24.904 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 30/34 2024-05-04T17:34:24.904 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-04T17:34:24.905 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-04T17:34:24.905 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:24.932 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 31/34 2024-05-04T17:34:34.174 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 31/34 2024-05-04T17:34:34.174 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-05-04T17:34:34.175 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-05-04T17:34:34.175 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-05-04T17:34:34.175 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-05-04T17:34:34.175 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-05-04T17:34:34.175 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-05-04T17:34:34.175 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-05-04T17:34:34.176 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:35.000 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 31/34 2024-05-04T17:34:35.000 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /sys 2024-05-04T17:34:35.000 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /proc 2024-05-04T17:34:35.000 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /mnt 2024-05-04T17:34:35.001 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /var/tmp 2024-05-04T17:34:35.001 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /home 2024-05-04T17:34:35.001 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /root 2024-05-04T17:34:35.001 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /tmp 2024-05-04T17:34:35.001 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:35.138 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 32/34 2024-05-04T17:34:35.164 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 32/34 2024-05-04T17:34:35.164 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:35.164 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-04T17:34:35.164 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-04T17:34:35.164 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-04T17:34:35.164 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:35.166 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-04T17:34:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-04T17:34:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-04T17:34:35.991 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 32/34 2024-05-04T17:34:36.016 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 32/34 2024-05-04T17:34:36.016 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:36.017 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-04T17:34:36.017 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-04T17:34:36.017 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-04T17:34:36.017 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:36.019 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-04T17:34:36.102 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-05-04T17:34:36.102 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-04T17:34:36.497 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-04T17:34:36.497 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 1/34 2024-05-04T17:34:36.497 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 2/34 2024-05-04T17:34:36.497 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 3/34 2024-05-04T17:34:36.497 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 4/34 2024-05-04T17:34:36.497 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 5/34 2024-05-04T17:34:36.497 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86_6 6/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 7/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2497.gf273489c.el9. 8/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-2497.gf273489c.el9.x8 9/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_64 10/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 11/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 12/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 13/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-04T17:34:36.498 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-04T17:34:36.499 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-04T17:34:36.499 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-04T17:34:36.499 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-04T17:34:36.499 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-04T17:34:36.499 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-04T17:34:36.499 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-04T17:34:36.499 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-04T17:34:36.499 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-04T17:34:36.499 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-04T17:34:36.499 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-04T17:34:36.500 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-2497.gf273489c.el9.x86_64 31/34 2024-05-04T17:34:36.500 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-04T17:34:36.500 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 33/34 2024-05-04T17:34:37.272 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-04T17:34:37.272 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:37.272 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-05-04T17:34:37.272 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.272 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.272 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:34:37.272 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-04T17:34:37.272 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.272 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.272 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-04T17:34:37.273 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:37.274 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:34:37.452 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-05-04T17:34:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 1/34 2024-05-04T17:34:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 2/34 2024-05-04T17:34:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 3/34 2024-05-04T17:34:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 4/34 2024-05-04T17:34:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 5/34 2024-05-04T17:34:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86_6 6/34 2024-05-04T17:34:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 7/34 2024-05-04T17:34:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2497.gf273489c.el9. 8/34 2024-05-04T17:34:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-common-2:19.0.0-2497.gf273489c.el9.x8 9/34 2024-05-04T17:34:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_64 10/34 2024-05-04T17:34:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 11/34 2024-05-04T17:34:37.453 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 12/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 13/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-05-04T17:34:37.454 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-05-04T17:34:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-05-04T17:34:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-05-04T17:34:37.455 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-05-04T17:34:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-05-04T17:34:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:19.0.0-2497.gf273489c.el9.x86_64 31/34 2024-05-04T17:34:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-05-04T17:34:37.456 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 33/34 2024-05-04T17:34:37.547 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-test 2024-05-04T17:34:38.112 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:30 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:34:38.237 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:34:38.238 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:34:38.238 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-04T17:34:38.238 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:34:38.238 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:34:38.238 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-2497.gf273489c.el9 ceph 48 M 2024-05-04T17:34:38.238 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-04T17:34:38.238 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-2497.gf273489c.el9 ceph 163 k 2024-05-04T17:34:38.238 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-04T17:34:38.238 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-04T17:34:38.238 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:38.238 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:34:38.238 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:34:38.239 INFO:teuthology.orchestra.run.smithi186.stdout:Install 4 Packages 2024-05-04T17:34:38.239 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:38.239 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 49 M 2024-05-04T17:34:38.239 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 207 M 2024-05-04T17:34:38.239 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:34:38.247 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-04T17:34:38.248 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-04T17:34:38.249 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-04T17:34:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-04T17:34:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-04T17:34:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: re2-1:20211101-3.el9.x86_64 2024-05-04T17:34:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-04T17:34:38.250 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:38.250 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:34:38.519 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-test 2024-05-04T17:34:38.758 INFO:teuthology.orchestra.run.smithi186.stdout:(1/4): libcephsqlite-19.0.0-2497.gf273489c.el9. 676 kB/s | 163 kB 00:00 2024-05-04T17:34:38.816 INFO:teuthology.orchestra.run.smithi186.stdout:(2/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.0 MB/s | 305 kB 00:00 2024-05-04T17:34:38.867 INFO:teuthology.orchestra.run.smithi186.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 585 kB/s | 64 kB 00:00 2024-05-04T17:34:39.059 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:33 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:34:39.184 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:34:39.184 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:34:39.184 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-05-04T17:34:39.184 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:34:39.185 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:34:39.185 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test x86_64 2:19.0.0-2497.gf273489c.el9 ceph 48 M 2024-05-04T17:34:39.185 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-05-04T17:34:39.185 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite x86_64 2:19.0.0-2497.gf273489c.el9 ceph 163 k 2024-05-04T17:34:39.185 INFO:teuthology.orchestra.run.smithi019.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-04T17:34:39.185 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-04T17:34:39.185 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:39.185 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:34:39.185 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:34:39.185 INFO:teuthology.orchestra.run.smithi019.stdout:Install 4 Packages 2024-05-04T17:34:39.185 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:39.186 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 49 M 2024-05-04T17:34:39.186 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 207 M 2024-05-04T17:34:39.186 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:34:39.484 INFO:teuthology.orchestra.run.smithi019.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 305 kB 00:00 2024-05-04T17:34:39.543 INFO:teuthology.orchestra.run.smithi019.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-04T17:34:40.007 INFO:teuthology.orchestra.run.smithi019.stdout:(3/4): libcephsqlite-19.0.0-2497.gf273489c.el9. 233 kB/s | 163 kB 00:00 2024-05-04T17:34:40.934 INFO:teuthology.orchestra.run.smithi186.stdout:(4/4): ceph-test-19.0.0-2497.gf273489c.el9.x86_ 20 MB/s | 48 MB 00:02 2024-05-04T17:34:40.937 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:34:40.938 INFO:teuthology.orchestra.run.smithi186.stdout:Total 18 MB/s | 49 MB 00:02 2024-05-04T17:34:40.964 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:34:40.985 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:34:40.985 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:34:41.350 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:34:41.350 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:34:41.867 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:34:41.935 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-04T17:34:41.984 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-04T17:34:42.011 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 3/4 2024-05-04T17:34:43.353 INFO:teuthology.orchestra.run.smithi019.stdout:(4/4): ceph-test-19.0.0-2497.gf273489c.el9.x86_ 12 MB/s | 48 MB 00:04 2024-05-04T17:34:43.354 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:34:43.355 INFO:teuthology.orchestra.run.smithi019.stdout:Total 12 MB/s | 49 MB 00:04 2024-05-04T17:34:43.379 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:34:43.399 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:34:43.399 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:34:43.782 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:34:43.783 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:34:44.324 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:34:44.397 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-05-04T17:34:44.436 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-05-04T17:34:44.464 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 3/4 2024-05-04T17:34:46.525 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 3/4 2024-05-04T17:34:46.602 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 4/4 2024-05-04T17:34:48.863 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 4/4 2024-05-04T17:34:48.863 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 1/4 2024-05-04T17:34:48.863 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 2/4 2024-05-04T17:34:48.863 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-04T17:34:49.169 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-04T17:34:49.170 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:49.170 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:34:49.170 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:49.170 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:49.170 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-04T17:34:49.170 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-04T17:34:49.170 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:49.170 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:34:49.266 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 3/4 2024-05-04T17:34:49.347 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 4/4 2024-05-04T17:34:49.417 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph 2024-05-04T17:34:49.945 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:41 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:34:50.079 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:34:50.080 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:34:50.080 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-04T17:34:50.081 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:34:50.081 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:34:50.081 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-2497.gf273489c.el9 ceph 6.6 k 2024-05-04T17:34:50.081 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-04T17:34:50.081 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-2497.gf273489c.el9 ceph 2.4 M 2024-05-04T17:34:50.081 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-2497.gf273489c.el9 ceph 1.5 M 2024-05-04T17:34:50.081 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 249 k 2024-05-04T17:34:50.081 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-2497.gf273489c.el9 ceph 4.7 M 2024-05-04T17:34:50.081 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-2497.gf273489c.el9 ceph 17 M 2024-05-04T17:34:50.081 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-04T17:34:50.081 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-04T17:34:50.081 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-04T17:34:50.082 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-04T17:34:50.083 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-04T17:34:50.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-04T17:34:50.084 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-04T17:34:50.084 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:50.084 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:34:50.084 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:34:50.084 INFO:teuthology.orchestra.run.smithi186.stdout:Install 36 Packages 2024-05-04T17:34:50.084 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:50.085 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 30 M 2024-05-04T17:34:50.085 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 107 M 2024-05-04T17:34:50.085 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:34:51.152 INFO:teuthology.orchestra.run.smithi186.stdout:(1/36): ceph-19.0.0-2497.gf273489c.el9.x86_64.r 12 kB/s | 6.6 kB 00:00 2024-05-04T17:34:51.468 INFO:teuthology.orchestra.run.smithi186.stdout:(2/36): ceph-mgr-19.0.0-2497.gf273489c.el9.x86_ 1.7 MB/s | 1.5 MB 00:00 2024-05-04T17:34:51.652 INFO:teuthology.orchestra.run.smithi186.stdout:(3/36): ceph-mds-19.0.0-2497.gf273489c.el9.x86_ 2.2 MB/s | 2.4 MB 00:01 2024-05-04T17:34:51.681 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 4/4 2024-05-04T17:34:51.682 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 1/4 2024-05-04T17:34:51.682 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 2/4 2024-05-04T17:34:51.682 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-05-04T17:34:51.777 INFO:teuthology.orchestra.run.smithi186.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2497.gf273 1.9 MB/s | 249 kB 00:00 2024-05-04T17:34:51.936 INFO:teuthology.orchestra.run.smithi186.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-05-04T17:34:51.963 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-05-04T17:34:51.964 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:51.964 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:34:51.964 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:51.964 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:34:51.964 INFO:teuthology.orchestra.run.smithi019.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-04T17:34:51.964 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-04T17:34:51.965 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:51.965 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:34:52.036 INFO:teuthology.orchestra.run.smithi186.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 12 MB/s | 1.2 MB 00:00 2024-05-04T17:34:52.086 INFO:teuthology.orchestra.run.smithi186.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-04T17:34:52.137 INFO:teuthology.orchestra.run.smithi186.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.7 MB/s | 135 kB 00:00 2024-05-04T17:34:52.187 INFO:teuthology.orchestra.run.smithi186.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-05-04T17:34:52.217 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph 2024-05-04T17:34:52.237 INFO:teuthology.orchestra.run.smithi186.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-05-04T17:34:52.329 INFO:teuthology.orchestra.run.smithi186.stdout:(11/36): ceph-mon-19.0.0-2497.gf273489c.el9.x86 4.0 MB/s | 4.7 MB 00:01 2024-05-04T17:34:52.438 INFO:teuthology.orchestra.run.smithi186.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 942 kB/s | 188 kB 00:00 2024-05-04T17:34:52.472 INFO:teuthology.orchestra.run.smithi186.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-05-04T17:34:52.488 INFO:teuthology.orchestra.run.smithi186.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 692 kB/s | 35 kB 00:00 2024-05-04T17:34:52.506 INFO:teuthology.orchestra.run.smithi186.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.5 MB/s | 51 kB 00:00 2024-05-04T17:34:52.522 INFO:teuthology.orchestra.run.smithi186.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-05-04T17:34:52.681 INFO:teuthology.orchestra.run.smithi186.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-05-04T17:34:52.745 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:46 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:34:52.823 INFO:teuthology.orchestra.run.smithi186.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 503 kB/s | 151 kB 00:00 2024-05-04T17:34:52.873 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:34:52.876 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:34:52.876 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-05-04T17:34:52.876 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:34:52.876 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:34:52.876 INFO:teuthology.orchestra.run.smithi019.stdout: ceph x86_64 2:19.0.0-2497.gf273489c.el9 ceph 6.6 k 2024-05-04T17:34:52.876 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-05-04T17:34:52.876 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds x86_64 2:19.0.0-2497.gf273489c.el9 ceph 2.4 M 2024-05-04T17:34:52.876 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr x86_64 2:19.0.0-2497.gf273489c.el9 ceph 1.5 M 2024-05-04T17:34:52.876 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 249 k 2024-05-04T17:34:52.876 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon x86_64 2:19.0.0-2497.gf273489c.el9 ceph 4.7 M 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd x86_64 2:19.0.0-2497.gf273489c.el9 ceph 17 M 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-05-04T17:34:52.877 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-04T17:34:52.878 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-04T17:34:52.879 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-04T17:34:52.879 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-04T17:34:52.879 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-04T17:34:52.879 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-04T17:34:52.879 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-04T17:34:52.879 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:52.879 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:34:52.879 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:34:52.879 INFO:teuthology.orchestra.run.smithi019.stdout:Install 36 Packages 2024-05-04T17:34:52.879 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:34:52.880 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 30 M 2024-05-04T17:34:52.880 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 107 M 2024-05-04T17:34:52.880 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:34:52.881 INFO:teuthology.orchestra.run.smithi186.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 217 kB/s | 43 kB 00:00 2024-05-04T17:34:53.457 INFO:teuthology.orchestra.run.smithi186.stdout:(20/36): ceph-osd-19.0.0-2497.gf273489c.el9.x86 8.4 MB/s | 17 MB 00:01 2024-05-04T17:34:53.482 INFO:teuthology.orchestra.run.smithi186.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 261 kB/s | 172 kB 00:00 2024-05-04T17:34:53.574 INFO:teuthology.orchestra.run.smithi186.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-05-04T17:34:53.599 INFO:teuthology.orchestra.run.smithi186.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 500 kB/s | 358 kB 00:00 2024-05-04T17:34:53.624 INFO:teuthology.orchestra.run.smithi186.stdout:(24/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 64 kB/s | 11 kB 00:00 2024-05-04T17:34:53.675 INFO:teuthology.orchestra.run.smithi186.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-05-04T17:34:53.708 INFO:teuthology.orchestra.run.smithi186.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 236 kB/s | 20 kB 00:00 2024-05-04T17:34:53.734 INFO:teuthology.orchestra.run.smithi186.stdout:(27/36): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-05-04T17:34:53.759 INFO:teuthology.orchestra.run.smithi186.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 553 kB/s | 46 kB 00:00 2024-05-04T17:34:53.834 INFO:teuthology.orchestra.run.smithi186.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 628 kB/s | 79 kB 00:00 2024-05-04T17:34:53.879 INFO:teuthology.orchestra.run.smithi019.stdout:(1/36): ceph-19.0.0-2497.gf273489c.el9.x86_64.r 25 kB/s | 6.6 kB 00:00 2024-05-04T17:34:53.918 INFO:teuthology.orchestra.run.smithi186.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-05-04T17:34:53.976 INFO:teuthology.orchestra.run.smithi186.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-05-04T17:34:54.002 INFO:teuthology.orchestra.run.smithi186.stdout:(32/36): python3-portend-3.1.0-2.el9.noarch.rpm 68 kB/s | 16 kB 00:00 2024-05-04T17:34:54.026 INFO:teuthology.orchestra.run.smithi186.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 328 kB/s | 36 kB 00:00 2024-05-04T17:34:54.060 INFO:teuthology.orchestra.run.smithi186.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-05-04T17:34:54.102 INFO:teuthology.orchestra.run.smithi186.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-05-04T17:34:54.202 INFO:teuthology.orchestra.run.smithi186.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.1 MB/s | 427 kB 00:00 2024-05-04T17:34:54.206 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:34:54.206 INFO:teuthology.orchestra.run.smithi186.stdout:Total 7.3 MB/s | 30 MB 00:04 2024-05-04T17:34:54.342 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:34:54.381 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:34:54.382 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:34:54.675 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:34:54.677 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:34:54.712 INFO:teuthology.orchestra.run.smithi019.stdout:(2/36): ceph-mgr-19.0.0-2497.gf273489c.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2024-05-04T17:34:55.229 INFO:teuthology.orchestra.run.smithi019.stdout:(3/36): ceph-mds-19.0.0-2497.gf273489c.el9.x86_ 1.5 MB/s | 2.4 MB 00:01 2024-05-04T17:34:55.338 INFO:teuthology.orchestra.run.smithi019.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2497.gf273 2.2 MB/s | 249 kB 00:00 2024-05-04T17:34:55.439 INFO:teuthology.orchestra.run.smithi019.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-05-04T17:34:55.564 INFO:teuthology.orchestra.run.smithi019.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 9.8 MB/s | 1.2 MB 00:00 2024-05-04T17:34:55.631 INFO:teuthology.orchestra.run.smithi019.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-05-04T17:34:55.806 INFO:teuthology.orchestra.run.smithi019.stdout:(8/36): ceph-mon-19.0.0-2497.gf273489c.el9.x86_ 2.4 MB/s | 4.7 MB 00:01 2024-05-04T17:34:55.865 INFO:teuthology.orchestra.run.smithi019.stdout:(9/36): python3-pycparser-2.20-6.el9.noarch.rpm 579 kB/s | 135 kB 00:00 2024-05-04T17:34:55.942 INFO:teuthology.orchestra.run.smithi019.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 940 kB/s | 125 kB 00:00 2024-05-04T17:34:55.948 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:34:55.965 INFO:teuthology.orchestra.run.smithi019.stdout:(11/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.1 MB/s | 215 kB 00:00 2024-05-04T17:34:56.022 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-04T17:34:56.024 INFO:teuthology.orchestra.run.smithi019.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-05-04T17:34:56.054 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-04T17:34:56.057 INFO:teuthology.orchestra.run.smithi019.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-05-04T17:34:56.082 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-04T17:34:56.099 INFO:teuthology.orchestra.run.smithi019.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 463 kB/s | 35 kB 00:00 2024-05-04T17:34:56.124 INFO:teuthology.orchestra.run.smithi019.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 763 kB/s | 51 kB 00:00 2024-05-04T17:34:56.153 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-04T17:34:56.175 INFO:teuthology.orchestra.run.smithi019.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 555 kB/s | 42 kB 00:00 2024-05-04T17:34:56.207 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-04T17:34:56.225 INFO:teuthology.orchestra.run.smithi019.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-05-04T17:34:56.239 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-04T17:34:56.331 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-04T17:34:56.403 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-04T17:34:56.450 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-04T17:34:56.485 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-04T17:34:56.515 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-04T17:34:56.534 INFO:teuthology.orchestra.run.smithi019.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 155 kB/s | 43 kB 00:00 2024-05-04T17:34:56.549 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-04T17:34:56.567 INFO:teuthology.orchestra.run.smithi019.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 385 kB/s | 151 kB 00:00 2024-05-04T17:34:56.583 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-04T17:34:56.623 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-04T17:34:56.658 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-04T17:34:56.693 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-04T17:34:56.709 INFO:teuthology.orchestra.run.smithi019.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 982 kB/s | 172 kB 00:00 2024-05-04T17:34:56.722 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-04T17:34:56.734 INFO:teuthology.orchestra.run.smithi019.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-05-04T17:34:56.785 INFO:teuthology.orchestra.run.smithi019.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-05-04T17:34:56.796 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-04T17:34:56.810 INFO:teuthology.orchestra.run.smithi019.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-05-04T17:34:56.852 INFO:teuthology.orchestra.run.smithi019.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-05-04T17:34:56.885 INFO:teuthology.orchestra.run.smithi019.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-05-04T17:34:56.919 INFO:teuthology.orchestra.run.smithi019.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-05-04T17:34:56.936 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-04T17:34:57.022 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-04T17:34:57.076 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-04T17:34:57.086 INFO:teuthology.orchestra.run.smithi019.stdout:(27/36): ceph-osd-19.0.0-2497.gf273489c.el9.x86 7.1 MB/s | 17 MB 00:02 2024-05-04T17:34:57.103 INFO:teuthology.orchestra.run.smithi019.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 213 kB/s | 46 kB 00:00 2024-05-04T17:34:57.114 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-04T17:34:57.128 INFO:teuthology.orchestra.run.smithi019.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 377 kB/s | 79 kB 00:00 2024-05-04T17:34:57.171 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-04T17:34:57.178 INFO:teuthology.orchestra.run.smithi019.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-05-04T17:34:57.212 INFO:teuthology.orchestra.run.smithi019.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-05-04T17:34:57.235 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-04T17:34:57.254 INFO:teuthology.orchestra.run.smithi019.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-05-04T17:34:57.321 INFO:teuthology.orchestra.run.smithi019.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-05-04T17:34:57.372 INFO:teuthology.orchestra.run.smithi019.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-05-04T17:34:57.388 INFO:teuthology.orchestra.run.smithi019.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 298 kB/s | 20 kB 00:00 2024-05-04T17:34:57.423 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-04T17:34:57.477 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-04T17:34:57.488 INFO:teuthology.orchestra.run.smithi019.stdout:(36/36): python3-pecan-1.4.2-3.el9.noarch.rpm 678 kB/s | 272 kB 00:00 2024-05-04T17:34:57.494 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:34:57.494 INFO:teuthology.orchestra.run.smithi019.stdout:Total 6.5 MB/s | 30 MB 00:04 2024-05-04T17:34:57.555 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-04T17:34:57.636 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:34:57.643 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-04T17:34:57.680 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:34:57.680 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:34:57.692 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-04T17:34:57.809 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-04T17:34:57.974 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2497.gf273489c.el9. 31/36 2024-05-04T17:34:57.994 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:34:57.995 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:34:58.002 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 32/36 2024-05-04T17:34:58.028 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 32/36 2024-05-04T17:34:58.028 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:58.028 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-04T17:34:58.028 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-04T17:34:58.028 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-04T17:34:58.028 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:59.274 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:34:59.333 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-05-04T17:34:59.366 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-05-04T17:34:59.394 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-05-04T17:34:59.462 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 33/36 2024-05-04T17:34:59.469 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-05-04T17:34:59.487 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 33/36 2024-05-04T17:34:59.487 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:59.487 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-04T17:34:59.487 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-04T17:34:59.487 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-04T17:34:59.488 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:59.518 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-05-04T17:34:59.551 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-05-04T17:34:59.647 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-05-04T17:34:59.725 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-05-04T17:34:59.771 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-05-04T17:34:59.805 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-05-04T17:34:59.835 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-05-04T17:34:59.869 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-05-04T17:34:59.903 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-05-04T17:34:59.937 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 34/36 2024-05-04T17:34:59.944 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-05-04T17:34:59.960 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 34/36 2024-05-04T17:34:59.961 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:34:59.961 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-04T17:34:59.961 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-04T17:34:59.961 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-04T17:34:59.961 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:34:59.987 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-05-04T17:35:00.021 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-05-04T17:35:00.050 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-05-04T17:35:00.128 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-05-04T17:35:00.180 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 35/36 2024-05-04T17:35:00.205 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 35/36 2024-05-04T17:35:00.205 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:35:00.205 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-04T17:35:00.205 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-04T17:35:00.205 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-04T17:35:00.205 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:00.267 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-2497.gf273489c.el9.x86_64 36/36 2024-05-04T17:35:00.281 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-05-04T17:35:00.362 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-05-04T17:35:00.422 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-05-04T17:35:00.459 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-05-04T17:35:00.520 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-05-04T17:35:00.592 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-05-04T17:35:00.790 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-05-04T17:35:00.840 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-05-04T17:35:01.115 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-05-04T17:35:01.203 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-05-04T17:35:01.239 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-05-04T17:35:01.362 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-05-04T17:35:01.527 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2497.gf273489c.el9. 31/36 2024-05-04T17:35:01.557 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 32/36 2024-05-04T17:35:01.584 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 32/36 2024-05-04T17:35:01.584 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:35:01.584 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-04T17:35:01.584 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-04T17:35:01.584 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-04T17:35:01.584 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:02.004 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-2:19.0.0-2497.gf273489c.el9.x86_64 36/36 2024-05-04T17:35:02.004 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-2497.gf273489c.el9.x86_64 1/36 2024-05-04T17:35:02.004 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 2/36 2024-05-04T17:35:02.004 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 3/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 4/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 5/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2497.gf273489c.el9. 6/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-04T17:35:02.005 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-04T17:35:02.006 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-04T17:35:02.006 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-04T17:35:02.006 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-04T17:35:02.006 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-04T17:35:02.006 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-04T17:35:02.006 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-04T17:35:02.006 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-04T17:35:02.006 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-04T17:35:02.006 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-04T17:35:02.006 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-04T17:35:02.007 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-04T17:35:02.007 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-04T17:35:02.007 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-04T17:35:02.007 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-04T17:35:02.007 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-04T17:35:02.008 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-04T17:35:02.008 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-04T17:35:02.840 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-04T17:35:02.841 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-04T17:35:02.842 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-04T17:35:02.843 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-04T17:35:02.843 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-04T17:35:02.843 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-04T17:35:02.843 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-04T17:35:02.843 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-04T17:35:02.843 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-04T17:35:02.843 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-04T17:35:02.843 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-04T17:35:02.843 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:02.843 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:03.108 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 33/36 2024-05-04T17:35:03.136 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 33/36 2024-05-04T17:35:03.136 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:35:03.136 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-04T17:35:03.136 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-04T17:35:03.136 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-04T17:35:03.136 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:03.201 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-base 2024-05-04T17:35:03.600 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 34/36 2024-05-04T17:35:03.625 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 34/36 2024-05-04T17:35:03.625 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:35:03.625 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-04T17:35:03.625 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-04T17:35:03.625 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-04T17:35:03.625 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:03.716 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:55 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:03.794 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:03.835 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:03.836 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-05-04T17:35:03.836 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:03.859 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 35/36 2024-05-04T17:35:03.884 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 35/36 2024-05-04T17:35:03.884 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:35:03.884 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-04T17:35:03.884 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-04T17:35:03.884 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-04T17:35:03.884 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:03.900 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install cephadm 2024-05-04T17:35:03.957 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-2:19.0.0-2497.gf273489c.el9.x86_64 36/36 2024-05-04T17:35:04.405 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:56 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:04.525 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:04.526 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:04.526 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-04T17:35:04.526 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:04.526 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:35:04.526 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 763 k 2024-05-04T17:35:04.526 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:04.526 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:35:04.526 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:04.526 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:35:04.526 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:04.527 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 763 k 2024-05-04T17:35:04.527 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 769 k 2024-05-04T17:35:04.527 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:35:05.205 INFO:teuthology.orchestra.run.smithi186.stdout:cephadm-19.0.0-2497.gf273489c.el9.noarch.rpm 1.1 MB/s | 763 kB 00:00 2024-05-04T17:35:05.205 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:05.205 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.1 MB/s | 763 kB 00:00 2024-05-04T17:35:05.206 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:35:05.211 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:35:05.211 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:35:05.217 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:35:05.217 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:35:05.289 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:35:05.590 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-2:19.0.0-2497.gf273489c.el9.x86_64 36/36 2024-05-04T17:35:05.590 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-2:19.0.0-2497.gf273489c.el9.x86_64 1/36 2024-05-04T17:35:05.590 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 2/36 2024-05-04T17:35:05.590 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 3/36 2024-05-04T17:35:05.590 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 4/36 2024-05-04T17:35:05.590 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 5/36 2024-05-04T17:35:05.590 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2497.gf273489c.el9. 6/36 2024-05-04T17:35:05.590 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-05-04T17:35:05.590 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-05-04T17:35:05.590 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-05-04T17:35:05.591 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-05-04T17:35:05.592 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-05-04T17:35:05.592 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-05-04T17:35:05.593 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-05-04T17:35:05.593 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-05-04T17:35:05.593 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-05-04T17:35:05.593 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-05-04T17:35:05.593 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-05-04T17:35:05.593 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-05-04T17:35:05.593 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-05-04T17:35:05.593 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-05-04T17:35:05.593 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-05-04T17:35:05.593 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-05-04T17:35:05.721 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-2497.gf273489c.el9.noarch 1/1 2024-05-04T17:35:05.807 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-2497.gf273489c.el9.noarch 1/1 2024-05-04T17:35:06.154 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-2497.gf273489c.el9.noarch 1/1 2024-05-04T17:35:06.368 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-2497.gf273489c.el9.noarch 1/1 2024-05-04T17:35:06.368 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:06.368 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:35:06.368 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:06.368 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:06.368 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:06.490 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-05-04T17:35:06.490 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:06.490 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:35:06.490 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: lua-5.4.4-4.el9.x86_64 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-04T17:35:06.491 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply-3.11-14.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-04T17:35:06.492 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-04T17:35:06.493 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-04T17:35:06.493 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-04T17:35:06.493 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-04T17:35:06.493 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-04T17:35:06.493 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-04T17:35:06.493 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-04T17:35:06.493 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:06.493 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:06.548 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-immutable-object-cache 2024-05-04T17:35:06.800 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base 2024-05-04T17:35:07.067 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:00:59 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:07.188 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:07.189 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:07.189 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:35:07.189 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:07.189 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:35:07.189 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2497.gf273489c.el9 ceph 145 k 2024-05-04T17:35:07.189 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:07.190 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:35:07.190 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:07.190 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:35:07.190 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:07.190 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 145 k 2024-05-04T17:35:07.190 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 435 k 2024-05-04T17:35:07.190 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:35:07.339 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:01 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:07.410 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-immutable-object-cache-19.0.0-2497.gf27348 656 kB/s | 145 kB 00:00 2024-05-04T17:35:07.411 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:07.411 INFO:teuthology.orchestra.run.smithi186.stdout:Total 651 kB/s | 145 kB 00:00 2024-05-04T17:35:07.411 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:35:07.419 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:35:07.419 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:35:07.420 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-base-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:07.463 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:07.464 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-05-04T17:35:07.465 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:07.475 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:35:07.475 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:35:07.534 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install cephadm 2024-05-04T17:35:07.622 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:35:07.711 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2497.gf273489c. 1/1 2024-05-04T17:35:07.735 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2497.gf273489c. 1/1 2024-05-04T17:35:07.736 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:35:07.736 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-04T17:35:07.736 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:08.064 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:02 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:08.189 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:08.190 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:08.190 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-05-04T17:35:08.190 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:08.190 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:35:08.190 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 763 k 2024-05-04T17:35:08.190 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:08.190 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:35:08.191 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:08.191 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:35:08.191 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:08.191 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 763 k 2024-05-04T17:35:08.191 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 769 k 2024-05-04T17:35:08.191 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:35:08.459 INFO:teuthology.orchestra.run.smithi019.stdout:cephadm-19.0.0-2497.gf273489c.el9.noarch.rpm 2.8 MB/s | 763 kB 00:00 2024-05-04T17:35:08.460 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:08.460 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.8 MB/s | 763 kB 00:00 2024-05-04T17:35:08.460 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:35:08.466 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:35:08.466 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:35:08.472 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:35:08.472 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:35:08.543 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:35:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2497.gf273489c. 1/1 2024-05-04T17:35:08.567 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:08.567 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:35:08.568 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:08.568 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:08.568 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:08.797 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr 2024-05-04T17:35:09.013 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:19.0.0-2497.gf273489c.el9.noarch 1/1 2024-05-04T17:35:09.107 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephadm-2:19.0.0-2497.gf273489c.el9.noarch 1/1 2024-05-04T17:35:09.315 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:01 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:09.395 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:09.426 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:19.0.0-2497.gf273489c.el9.noarch 1/1 2024-05-04T17:35:09.438 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:09.439 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-05-04T17:35:09.439 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:09.504 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-dashboard 2024-05-04T17:35:09.674 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephadm-2:19.0.0-2497.gf273489c.el9.noarch 1/1 2024-05-04T17:35:09.674 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:09.675 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:35:09.675 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:09.675 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:09.675 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:09.874 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-immutable-object-cache 2024-05-04T17:35:10.015 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:02 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:10.136 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:10.137 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:10.137 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-04T17:35:10.137 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:10.137 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:35:10.137 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 3.5 M 2024-05-04T17:35:10.137 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-04T17:35:10.137 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-04T17:35:10.137 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 32 k 2024-05-04T17:35:10.138 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 15 k 2024-05-04T17:35:10.138 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-04T17:35:10.138 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-04T17:35:10.138 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-04T17:35:10.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-04T17:35:10.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-04T17:35:10.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-04T17:35:10.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-04T17:35:10.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-04T17:35:10.138 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:10.138 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:35:10.138 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:10.139 INFO:teuthology.orchestra.run.smithi186.stdout:Install 12 Packages 2024-05-04T17:35:10.139 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:10.139 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 8.6 M 2024-05-04T17:35:10.139 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 103 M 2024-05-04T17:35:10.139 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:35:10.402 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:04 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:10.527 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:10.527 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:10.527 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:35:10.528 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:10.528 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:35:10.528 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2497.gf273489c.el9 ceph 145 k 2024-05-04T17:35:10.528 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:10.528 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:35:10.528 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:10.528 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:35:10.528 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:10.529 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 145 k 2024-05-04T17:35:10.529 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 435 k 2024-05-04T17:35:10.529 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:35:11.033 INFO:teuthology.orchestra.run.smithi186.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2497.gf2 77 kB/s | 32 kB 00:00 2024-05-04T17:35:11.093 INFO:teuthology.orchestra.run.smithi186.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2497.gf27 33 kB/s | 15 kB 00:00 2024-05-04T17:35:11.157 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-immutable-object-cache-19.0.0-2497.gf27348 230 kB/s | 145 kB 00:00 2024-05-04T17:35:11.158 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:11.158 INFO:teuthology.orchestra.run.smithi019.stdout:Total 230 kB/s | 145 kB 00:00 2024-05-04T17:35:11.158 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:35:11.167 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:35:11.167 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:35:11.229 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:35:11.250 INFO:teuthology.orchestra.run.smithi186.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.7 MB/s | 1.0 MB 00:00 2024-05-04T17:35:11.275 INFO:teuthology.orchestra.run.smithi186.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-05-04T17:35:11.382 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:35:11.459 INFO:teuthology.orchestra.run.smithi186.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.0 MB/s | 863 kB 00:00 2024-05-04T17:35:11.459 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2497.gf273489c. 1/1 2024-05-04T17:35:11.484 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2497.gf273489c. 1/1 2024-05-04T17:35:11.484 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:35:11.485 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-04T17:35:11.485 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:11.617 INFO:teuthology.orchestra.run.smithi186.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-05-04T17:35:11.642 INFO:teuthology.orchestra.run.smithi186.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2024-05-04T17:35:11.793 INFO:teuthology.orchestra.run.smithi186.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 14 MB/s | 2.0 MB 00:00 2024-05-04T17:35:11.818 INFO:teuthology.orchestra.run.smithi186.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 825 kB/s | 144 kB 00:00 2024-05-04T17:35:11.860 INFO:teuthology.orchestra.run.smithi186.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 462 kB/s | 31 kB 00:00 2024-05-04T17:35:11.877 INFO:teuthology.orchestra.run.smithi186.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-05-04T17:35:12.398 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2497.gf273489c. 1/1 2024-05-04T17:35:12.398 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:12.398 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:35:12.399 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:12.399 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:12.399 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:12.486 INFO:teuthology.orchestra.run.smithi186.stdout:(12/12): ceph-mgr-dashboard-19.0.0-2497.gf27348 1.9 MB/s | 3.5 MB 00:01 2024-05-04T17:35:12.489 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:12.489 INFO:teuthology.orchestra.run.smithi186.stdout:Total 3.7 MB/s | 8.6 MB 00:02 2024-05-04T17:35:12.539 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:35:12.556 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:35:12.557 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:35:12.603 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr 2024-05-04T17:35:12.710 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:35:12.711 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:35:13.133 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:07 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:13.216 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-mgr-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:13.232 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:35:13.260 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:13.261 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-05-04T17:35:13.261 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:13.332 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-dashboard 2024-05-04T17:35:13.336 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-04T17:35:13.371 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-04T17:35:13.427 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-04T17:35:13.456 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-04T17:35:13.534 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-04T17:35:13.659 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-04T17:35:13.752 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-04T17:35:13.803 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-04T17:35:13.831 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-04T17:35:13.863 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2497.gf273489c.el9 10/12 2024-05-04T17:35:13.865 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:07 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:13.991 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:13.992 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:13.992 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-05-04T17:35:13.992 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:13.992 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:35:13.992 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 3.5 M 2024-05-04T17:35:13.992 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-05-04T17:35:13.992 INFO:teuthology.orchestra.run.smithi019.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-04T17:35:13.993 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 32 k 2024-05-04T17:35:13.993 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 15 k 2024-05-04T17:35:13.993 INFO:teuthology.orchestra.run.smithi019.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-04T17:35:13.993 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-04T17:35:13.993 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-04T17:35:13.993 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-04T17:35:13.993 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-04T17:35:13.993 INFO:teuthology.orchestra.run.smithi019.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-04T17:35:13.993 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-04T17:35:13.993 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-04T17:35:13.993 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:13.993 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:35:13.994 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:13.994 INFO:teuthology.orchestra.run.smithi019.stdout:Install 12 Packages 2024-05-04T17:35:13.994 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:13.994 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 8.6 M 2024-05-04T17:35:13.994 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 103 M 2024-05-04T17:35:13.995 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:35:14.511 INFO:teuthology.orchestra.run.smithi019.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2497.gf27 73 kB/s | 15 kB 00:00 2024-05-04T17:35:14.545 INFO:teuthology.orchestra.run.smithi019.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2497.gf2 130 kB/s | 32 kB 00:00 2024-05-04T17:35:14.754 INFO:teuthology.orchestra.run.smithi019.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-05-04T17:35:14.787 INFO:teuthology.orchestra.run.smithi019.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.7 MB/s | 1.0 MB 00:00 2024-05-04T17:35:14.888 INFO:teuthology.orchestra.run.smithi019.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 6.3 MB/s | 863 kB 00:00 2024-05-04T17:35:14.926 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2497.gf273489c.el 11/12 2024-05-04T17:35:14.964 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.noa 12/12 2024-05-04T17:35:15.296 INFO:teuthology.orchestra.run.smithi019.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 51 kB/s | 19 kB 00:00 2024-05-04T17:35:15.505 INFO:teuthology.orchestra.run.smithi019.stdout:(7/12): ceph-mgr-dashboard-19.0.0-2497.gf273489 2.9 MB/s | 3.5 MB 00:01 2024-05-04T17:35:15.680 INFO:teuthology.orchestra.run.smithi019.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 614 kB/s | 548 kB 00:00 2024-05-04T17:35:15.847 INFO:teuthology.orchestra.run.smithi019.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 3.7 MB/s | 2.0 MB 00:00 2024-05-04T17:35:15.872 INFO:teuthology.orchestra.run.smithi019.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 394 kB/s | 144 kB 00:00 2024-05-04T17:35:15.897 INFO:teuthology.orchestra.run.smithi019.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 142 kB/s | 31 kB 00:00 2024-05-04T17:35:15.956 INFO:teuthology.orchestra.run.smithi019.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-05-04T17:35:15.960 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:15.960 INFO:teuthology.orchestra.run.smithi019.stdout:Total 4.4 MB/s | 8.6 MB 00:01 2024-05-04T17:35:16.012 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:35:16.031 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:35:16.031 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:35:16.196 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:35:16.196 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:35:16.278 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.noa 12/12 2024-05-04T17:35:16.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2497.gf273489c.el 1/12 2024-05-04T17:35:16.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.noa 2/12 2024-05-04T17:35:16.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2497.gf273489c.el9 3/12 2024-05-04T17:35:16.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-04T17:35:16.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-04T17:35:16.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-04T17:35:16.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-04T17:35:16.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-04T17:35:16.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-04T17:35:16.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-04T17:35:16.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-04T17:35:16.566 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-04T17:35:16.566 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:16.566 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:35:16.566 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-04T17:35:16.566 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:16.566 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:16.566 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:16.566 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-04T17:35:16.566 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-04T17:35:16.567 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-04T17:35:16.567 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-04T17:35:16.567 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-04T17:35:16.567 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-04T17:35:16.567 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-04T17:35:16.567 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-04T17:35:16.567 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:16.567 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:16.750 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:35:16.806 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-04T17:35:16.817 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-05-04T17:35:16.852 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-05-04T17:35:16.899 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-05-04T17:35:16.936 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-05-04T17:35:17.017 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-05-04T17:35:17.160 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-05-04T17:35:17.252 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-05-04T17:35:17.300 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-05-04T17:35:17.312 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:09 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:17.327 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-05-04T17:35:17.360 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2497.gf273489c.el9 10/12 2024-05-04T17:35:17.433 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:17.434 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-05-04T17:35:17.434 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-04T17:35:17.434 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-05-04T17:35:17.434 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:35:17.434 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 7.4 M 2024-05-04T17:35:17.434 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-04T17:35:17.434 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-04T17:35:17.434 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-04T17:35:17.434 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout:Install 12 Packages 2024-05-04T17:35:17.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:17.436 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 38 M 2024-05-04T17:35:17.436 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 209 M 2024-05-04T17:35:17.436 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:35:17.843 INFO:teuthology.orchestra.run.smithi186.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 767 kB/s | 195 kB 00:00 2024-05-04T17:35:17.869 INFO:teuthology.orchestra.run.smithi186.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-05-04T17:35:17.944 INFO:teuthology.orchestra.run.smithi186.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 326 kB/s | 33 kB 00:00 2024-05-04T17:35:17.994 INFO:teuthology.orchestra.run.smithi186.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 363 kB/s | 18 kB 00:00 2024-05-04T17:35:18.036 INFO:teuthology.orchestra.run.smithi186.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 849 kB/s | 35 kB 00:00 2024-05-04T17:35:18.459 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2497.gf273489c.el 11/12 2024-05-04T17:35:18.494 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.noa 12/12 2024-05-04T17:35:18.945 INFO:teuthology.orchestra.run.smithi186.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.8 MB/s | 3.0 MB 00:01 2024-05-04T17:35:19.053 INFO:teuthology.orchestra.run.smithi186.stdout:(7/12): ceph-mgr-diskprediction-local-19.0.0-24 5.1 MB/s | 7.4 MB 00:01 2024-05-04T17:35:19.103 INFO:teuthology.orchestra.run.smithi186.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.5 MB/s | 244 kB 00:00 2024-05-04T17:35:19.220 INFO:teuthology.orchestra.run.smithi186.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-05-04T17:35:19.563 INFO:teuthology.orchestra.run.smithi186.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.1 MB/s | 4.7 MB 00:01 2024-05-04T17:35:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.noa 12/12 2024-05-04T17:35:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2497.gf273489c.el 1/12 2024-05-04T17:35:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.noa 2/12 2024-05-04T17:35:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2497.gf273489c.el9 3/12 2024-05-04T17:35:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-05-04T17:35:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-05-04T17:35:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-05-04T17:35:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-05-04T17:35:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-05-04T17:35:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-05-04T17:35:19.851 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-05-04T17:35:19.852 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-05-04T17:35:20.165 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-05-04T17:35:20.165 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:20.165 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:35:20.165 INFO:teuthology.orchestra.run.smithi019.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-04T17:35:20.165 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:20.165 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:20.165 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:20.166 INFO:teuthology.orchestra.run.smithi019.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-04T17:35:20.166 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-04T17:35:20.166 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-04T17:35:20.166 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-04T17:35:20.166 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-04T17:35:20.166 INFO:teuthology.orchestra.run.smithi019.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-04T17:35:20.166 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-04T17:35:20.166 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-04T17:35:20.166 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:20.166 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:20.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-diskprediction-local 2024-05-04T17:35:20.646 INFO:teuthology.orchestra.run.smithi186.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.2 MB/s | 5.1 MB 00:01 2024-05-04T17:35:20.922 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:14 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:21.051 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 7.4 M 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-04T17:35:21.052 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-04T17:35:21.053 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-04T17:35:21.053 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-04T17:35:21.053 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-04T17:35:21.053 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:21.053 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:35:21.053 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================= 2024-05-04T17:35:21.053 INFO:teuthology.orchestra.run.smithi019.stdout:Install 12 Packages 2024-05-04T17:35:21.053 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:21.054 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 38 M 2024-05-04T17:35:21.054 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 209 M 2024-05-04T17:35:21.054 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:35:21.507 INFO:teuthology.orchestra.run.smithi019.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 735 kB/s | 195 kB 00:00 2024-05-04T17:35:21.543 INFO:teuthology.orchestra.run.smithi019.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-05-04T17:35:21.921 INFO:teuthology.orchestra.run.smithi186.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.0 MB/s | 16 MB 00:02 2024-05-04T17:35:21.922 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:21.922 INFO:teuthology.orchestra.run.smithi186.stdout:Total 8.5 MB/s | 38 MB 00:04 2024-05-04T17:35:22.100 INFO:teuthology.orchestra.run.smithi019.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 55 kB/s | 33 kB 00:00 2024-05-04T17:35:22.272 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:35:22.298 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:35:22.299 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:35:22.466 INFO:teuthology.orchestra.run.smithi019.stdout:(4/12): ceph-mgr-diskprediction-local-19.0.0-24 6.0 MB/s | 7.4 MB 00:01 2024-05-04T17:35:22.667 INFO:teuthology.orchestra.run.smithi019.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 32 kB/s | 18 kB 00:00 2024-05-04T17:35:22.706 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:35:22.706 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:35:22.733 INFO:teuthology.orchestra.run.smithi019.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.5 MB/s | 3.0 MB 00:01 2024-05-04T17:35:23.167 INFO:teuthology.orchestra.run.smithi019.stdout:(7/12): openblas-0.3.21-2.el9.x86_64.rpm 50 kB/s | 35 kB 00:00 2024-05-04T17:35:23.500 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:35:23.550 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-04T17:35:23.577 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-04T17:35:23.675 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-04T17:35:23.728 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-04T17:35:23.793 INFO:teuthology.orchestra.run.smithi019.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 236 kB/s | 244 kB 00:01 2024-05-04T17:35:23.907 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-04T17:35:23.944 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-04T17:35:24.021 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-04T17:35:24.234 INFO:teuthology.orchestra.run.smithi019.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 3.0 MB/s | 4.7 MB 00:01 2024-05-04T17:35:24.667 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-04T17:35:24.783 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-04T17:35:24.784 INFO:teuthology.orchestra.run.smithi019.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.2 MB/s | 5.1 MB 00:01 2024-05-04T17:35:24.901 INFO:teuthology.orchestra.run.smithi019.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 226 kB/s | 250 kB 00:01 2024-05-04T17:35:26.010 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-04T17:35:27.221 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-04T17:35:27.253 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2497.gf2734 12/12 2024-05-04T17:35:27.444 INFO:teuthology.orchestra.run.smithi019.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.1 MB/s | 16 MB 00:03 2024-05-04T17:35:27.446 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:27.446 INFO:teuthology.orchestra.run.smithi019.stdout:Total 5.9 MB/s | 38 MB 00:06 2024-05-04T17:35:27.810 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:35:27.838 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:35:27.838 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:35:28.274 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:35:28.274 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:35:29.130 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:35:29.190 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-05-04T17:35:29.234 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-05-04T17:35:29.350 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-05-04T17:35:29.385 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-05-04T17:35:29.562 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-05-04T17:35:29.592 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-05-04T17:35:29.670 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-05-04T17:35:29.843 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2497.gf2734 12/12 2024-05-04T17:35:29.843 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2497.gf2734 1/12 2024-05-04T17:35:29.843 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-04T17:35:29.844 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-04T17:35:29.844 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-04T17:35:29.844 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-04T17:35:29.844 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-04T17:35:29.844 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-04T17:35:29.844 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-04T17:35:29.844 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-04T17:35:29.844 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-04T17:35:29.844 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-04T17:35:30.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-04T17:35:30.236 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:30.236 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:35:30.236 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:30.236 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-04T17:35:30.236 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-04T17:35:30.236 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-04T17:35:30.236 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-04T17:35:30.237 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-04T17:35:30.237 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-04T17:35:30.237 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-04T17:35:30.237 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-04T17:35:30.237 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-04T17:35:30.237 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-04T17:35:30.238 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-04T17:35:30.238 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:30.238 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:30.371 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-05-04T17:35:30.494 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-05-04T17:35:30.588 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-rook 2024-05-04T17:35:31.118 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:23 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:31.237 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:31.239 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:31.239 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-04T17:35:31.239 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:31.239 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:35:31.239 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 49 k 2024-05-04T17:35:31.239 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-04T17:35:31.239 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-04T17:35:31.239 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-04T17:35:31.239 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-04T17:35:31.239 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-04T17:35:31.239 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-04T17:35:31.240 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-04T17:35:31.240 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-04T17:35:31.240 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-04T17:35:31.240 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-04T17:35:31.240 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-04T17:35:31.240 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-04T17:35:31.240 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-04T17:35:31.240 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:31.240 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:35:31.240 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:31.240 INFO:teuthology.orchestra.run.smithi186.stdout:Install 13 Packages 2024-05-04T17:35:31.240 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:31.241 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 2.2 M 2024-05-04T17:35:31.241 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 26 M 2024-05-04T17:35:31.241 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:35:31.825 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-05-04T17:35:31.862 INFO:teuthology.orchestra.run.smithi186.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 138 kB/s | 26 kB 00:00 2024-05-04T17:35:31.879 INFO:teuthology.orchestra.run.smithi186.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 95 kB/s | 19 kB 00:00 2024-05-04T17:35:31.954 INFO:teuthology.orchestra.run.smithi186.stdout:(3/13): ceph-mgr-rook-19.0.0-2497.gf273489c.el9 177 kB/s | 49 kB 00:00 2024-05-04T17:35:31.996 INFO:teuthology.orchestra.run.smithi186.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-05-04T17:35:32.046 INFO:teuthology.orchestra.run.smithi186.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-05-04T17:35:32.088 INFO:teuthology.orchestra.run.smithi186.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 786 kB/s | 54 kB 00:00 2024-05-04T17:35:32.205 INFO:teuthology.orchestra.run.smithi186.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-05-04T17:35:32.389 INFO:teuthology.orchestra.run.smithi186.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 94 kB/s | 32 kB 00:00 2024-05-04T17:35:32.417 INFO:teuthology.orchestra.run.smithi186.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 46 kB/s | 15 kB 00:00 2024-05-04T17:35:32.606 INFO:teuthology.orchestra.run.smithi186.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-05-04T17:35:32.715 INFO:teuthology.orchestra.run.smithi186.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 404 kB/s | 206 kB 00:00 2024-05-04T17:35:32.765 INFO:teuthology.orchestra.run.smithi186.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-05-04T17:35:32.922 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-05-04T17:35:32.950 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2497.gf2734 12/12 2024-05-04T17:35:33.082 INFO:teuthology.orchestra.run.smithi186.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-05-04T17:35:33.088 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:33.088 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-05-04T17:35:33.151 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:35:33.166 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:35:33.166 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:35:33.252 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:35:33.253 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:35:33.759 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:35:33.939 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-04T17:35:34.000 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-04T17:35:34.056 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-04T17:35:34.093 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-04T17:35:34.134 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-04T17:35:34.228 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-04T17:35:34.330 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-04T17:35:34.377 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-04T17:35:35.091 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-04T17:35:35.150 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-04T17:35:35.176 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-04T17:35:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-04T17:35:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 13/13 2024-05-04T17:35:35.568 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2497.gf2734 12/12 2024-05-04T17:35:35.568 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2497.gf2734 1/12 2024-05-04T17:35:35.568 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-05-04T17:35:35.568 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-05-04T17:35:35.569 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-05-04T17:35:35.569 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-05-04T17:35:35.569 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-05-04T17:35:35.569 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-05-04T17:35:35.569 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-05-04T17:35:35.569 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-05-04T17:35:35.569 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-05-04T17:35:35.569 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-05-04T17:35:35.956 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 13/13 2024-05-04T17:35:35.956 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 1/13 2024-05-04T17:35:35.956 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-04T17:35:35.956 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-04T17:35:35.956 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-04T17:35:35.956 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-04T17:35:35.956 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-04T17:35:35.956 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-04T17:35:35.957 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-04T17:35:35.957 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-04T17:35:35.957 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-04T17:35:35.957 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-04T17:35:35.957 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-04T17:35:35.976 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-05-04T17:35:35.976 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:35.976 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:35:35.976 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:35.976 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-04T17:35:35.976 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-04T17:35:35.977 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-04T17:35:35.977 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-04T17:35:35.977 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-04T17:35:35.977 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-04T17:35:35.977 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-04T17:35:35.977 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-04T17:35:35.977 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-04T17:35:35.977 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-04T17:35:35.977 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-04T17:35:35.977 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:35.977 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:36.263 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-04T17:35:36.263 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:36.263 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:35:36.263 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:36.263 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-04T17:35:36.263 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-04T17:35:36.263 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-04T17:35:36.263 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-04T17:35:36.263 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-04T17:35:36.264 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-04T17:35:36.264 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-04T17:35:36.264 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-04T17:35:36.264 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-04T17:35:36.264 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-04T17:35:36.264 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-04T17:35:36.264 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-04T17:35:36.264 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:36.264 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:36.353 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-rook 2024-05-04T17:35:36.488 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-cephadm 2024-05-04T17:35:36.890 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:30 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:36.997 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:28 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:37.014 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:37.015 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 49 k 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-04T17:35:37.016 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-04T17:35:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-04T17:35:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-04T17:35:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-04T17:35:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:37.017 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:35:37.017 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:37.017 INFO:teuthology.orchestra.run.smithi019.stdout:Install 13 Packages 2024-05-04T17:35:37.017 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:37.018 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.2 M 2024-05-04T17:35:37.018 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 26 M 2024-05-04T17:35:37.018 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:35:37.116 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:37.117 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:37.117 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-04T17:35:37.117 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:37.117 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:35:37.117 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 142 k 2024-05-04T17:35:37.117 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-04T17:35:37.117 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-04T17:35:37.117 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-04T17:35:37.117 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-04T17:35:37.117 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-04T17:35:37.118 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-04T17:35:37.118 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:37.118 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:35:37.118 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:37.118 INFO:teuthology.orchestra.run.smithi186.stdout:Install 6 Packages 2024-05-04T17:35:37.118 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:37.119 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 7.0 M 2024-05-04T17:35:37.119 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 33 M 2024-05-04T17:35:37.119 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:35:37.226 INFO:teuthology.orchestra.run.smithi019.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 338 kB/s | 26 kB 00:00 2024-05-04T17:35:37.251 INFO:teuthology.orchestra.run.smithi019.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 190 kB/s | 19 kB 00:00 2024-05-04T17:35:37.318 INFO:teuthology.orchestra.run.smithi019.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.4 MB/s | 159 kB 00:00 2024-05-04T17:35:37.351 INFO:teuthology.orchestra.run.smithi019.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-05-04T17:35:37.377 INFO:teuthology.orchestra.run.smithi019.stdout:(5/13): ceph-mgr-rook-19.0.0-2497.gf273489c.el9 217 kB/s | 49 kB 00:00 2024-05-04T17:35:37.427 INFO:teuthology.orchestra.run.smithi019.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 279 kB 00:00 2024-05-04T17:35:37.452 INFO:teuthology.orchestra.run.smithi019.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 533 kB/s | 54 kB 00:00 2024-05-04T17:35:37.477 INFO:teuthology.orchestra.run.smithi019.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-05-04T17:35:37.511 INFO:teuthology.orchestra.run.smithi019.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 179 kB/s | 15 kB 00:00 2024-05-04T17:35:37.593 INFO:teuthology.orchestra.run.smithi186.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-05-04T17:35:37.594 INFO:teuthology.orchestra.run.smithi019.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 709 kB/s | 59 kB 00:00 2024-05-04T17:35:37.620 INFO:teuthology.orchestra.run.smithi019.stdout:(11/13): python3-google-auth-2.29.0-1.el9.noarc 1.2 MB/s | 206 kB 00:00 2024-05-04T17:35:37.653 INFO:teuthology.orchestra.run.smithi019.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-05-04T17:35:37.704 INFO:teuthology.orchestra.run.smithi019.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.5 MB/s | 1.0 MB 00:00 2024-05-04T17:35:37.704 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:37.705 INFO:teuthology.orchestra.run.smithi019.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-05-04T17:35:37.760 INFO:teuthology.orchestra.run.smithi186.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2497.gf273489c.e 433 kB/s | 142 kB 00:00 2024-05-04T17:35:37.769 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:35:37.784 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:35:37.784 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:35:37.852 INFO:teuthology.orchestra.run.smithi186.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-05-04T17:35:37.880 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:35:37.880 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:35:38.019 INFO:teuthology.orchestra.run.smithi186.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-05-04T17:35:38.044 INFO:teuthology.orchestra.run.smithi186.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 203 kB/s | 58 kB 00:00 2024-05-04T17:35:38.069 INFO:teuthology.orchestra.run.smithi186.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 235 kB/s | 51 kB 00:00 2024-05-04T17:35:38.071 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:38.071 INFO:teuthology.orchestra.run.smithi186.stdout:Total 7.3 MB/s | 7.0 MB 00:00 2024-05-04T17:35:38.140 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:35:38.152 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:35:38.152 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:35:38.248 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:35:38.248 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:35:38.403 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:35:38.570 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-05-04T17:35:38.578 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:35:38.632 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-05-04T17:35:38.688 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-05-04T17:35:38.724 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-05-04T17:35:38.742 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-04T17:35:38.765 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-05-04T17:35:38.796 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-04T17:35:38.864 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-05-04T17:35:38.974 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-05-04T17:35:39.035 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-05-04T17:35:39.330 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-04T17:35:39.420 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-04T17:35:39.486 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-04T17:35:39.517 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noarch 6/6 2024-05-04T17:35:39.812 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-05-04T17:35:39.856 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-05-04T17:35:39.889 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-05-04T17:35:39.938 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-05-04T17:35:39.967 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 13/13 2024-05-04T17:35:40.244 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noarch 6/6 2024-05-04T17:35:40.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noarch 1/6 2024-05-04T17:35:40.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-04T17:35:40.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-04T17:35:40.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-04T17:35:40.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-04T17:35:40.496 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-04T17:35:40.496 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:40.496 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:35:40.496 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:40.496 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-04T17:35:40.496 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-04T17:35:40.496 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-04T17:35:40.496 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-04T17:35:40.496 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-04T17:35:40.496 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:40.496 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:40.629 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 13/13 2024-05-04T17:35:40.629 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 1/13 2024-05-04T17:35:40.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-05-04T17:35:40.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-05-04T17:35:40.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-05-04T17:35:40.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-05-04T17:35:40.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-05-04T17:35:40.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-05-04T17:35:40.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-05-04T17:35:40.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-05-04T17:35:40.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-05-04T17:35:40.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-05-04T17:35:40.630 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-05-04T17:35:40.743 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-fuse 2024-05-04T17:35:40.955 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-05-04T17:35:40.955 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:40.955 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:35:40.955 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:40.955 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-04T17:35:40.956 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-04T17:35:40.956 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-04T17:35:40.956 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-04T17:35:40.956 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-04T17:35:40.956 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-04T17:35:40.956 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-04T17:35:40.956 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-04T17:35:40.956 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-04T17:35:40.956 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-04T17:35:40.956 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-04T17:35:40.956 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-04T17:35:40.956 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:40.957 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:41.178 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-cephadm 2024-05-04T17:35:41.260 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:33 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:41.380 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:41.380 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:41.381 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-04T17:35:41.381 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:41.381 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:35:41.381 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-2497.gf273489c.el9 ceph 889 k 2024-05-04T17:35:41.381 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-04T17:35:41.381 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-04T17:35:41.381 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:41.381 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:35:41.381 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:41.381 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-05-04T17:35:41.381 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:41.382 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 970 k 2024-05-04T17:35:41.382 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 2.8 M 2024-05-04T17:35:41.382 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:35:41.612 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 697 kB/s | 80 kB 00:00 2024-05-04T17:35:41.717 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:35 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:41.842 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:41.843 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:41.843 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-05-04T17:35:41.843 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:41.843 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:35:41.843 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 142 k 2024-05-04T17:35:41.844 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-05-04T17:35:41.844 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-04T17:35:41.844 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-04T17:35:41.844 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-04T17:35:41.844 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-04T17:35:41.844 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-04T17:35:41.844 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:41.844 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:35:41.844 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:41.844 INFO:teuthology.orchestra.run.smithi019.stdout:Install 6 Packages 2024-05-04T17:35:41.844 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:41.845 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 7.0 M 2024-05-04T17:35:41.845 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 33 M 2024-05-04T17:35:41.845 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:35:42.037 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): ceph-fuse-19.0.0-2497.gf273489c.el9.x86_ 1.6 MB/s | 889 kB 00:00 2024-05-04T17:35:42.038 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:42.038 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.4 MB/s | 970 kB 00:00 2024-05-04T17:35:42.050 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:35:42.072 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:35:42.072 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:35:42.150 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:35:42.150 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:35:42.274 INFO:teuthology.orchestra.run.smithi019.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-05-04T17:35:42.331 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:35:42.357 INFO:teuthology.orchestra.run.smithi019.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2497.gf273489c.e 640 kB/s | 142 kB 00:00 2024-05-04T17:35:42.441 INFO:teuthology.orchestra.run.smithi019.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 6.0 MB 00:00 2024-05-04T17:35:42.482 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-04T17:35:42.552 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 2/2 2024-05-04T17:35:42.574 INFO:teuthology.orchestra.run.smithi019.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 266 kB/s | 58 kB 00:00 2024-05-04T17:35:42.600 INFO:teuthology.orchestra.run.smithi019.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 321 kB/s | 51 kB 00:00 2024-05-04T17:35:42.633 INFO:teuthology.orchestra.run.smithi019.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-05-04T17:35:42.634 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:42.634 INFO:teuthology.orchestra.run.smithi019.stdout:Total 8.8 MB/s | 7.0 MB 00:00 2024-05-04T17:35:42.708 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:35:42.719 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:35:42.719 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:35:42.822 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:35:42.822 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:35:43.140 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:35:43.273 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 2/2 2024-05-04T17:35:43.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 1/2 2024-05-04T17:35:43.313 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-05-04T17:35:43.377 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-05-04T17:35:43.529 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-04T17:35:43.529 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:43.529 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:35:43.529 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-04T17:35:43.529 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:43.529 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:43.792 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-volume 2024-05-04T17:35:43.932 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-05-04T17:35:44.015 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-05-04T17:35:44.087 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-05-04T17:35:44.114 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noarch 6/6 2024-05-04T17:35:44.319 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:36 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:44.446 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:44.447 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:44.447 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 264 k 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 341 k 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 1.5 M 2024-05-04T17:35:44.448 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:35:44.818 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): ceph-volume-19.0.0-2497.gf273489c.el9.no 1.0 MB/s | 264 kB 00:00 2024-05-04T17:35:44.827 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noarch 6/6 2024-05-04T17:35:44.827 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noarch 1/6 2024-05-04T17:35:44.827 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-05-04T17:35:44.827 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-05-04T17:35:44.827 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-05-04T17:35:44.827 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-05-04T17:35:45.103 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-05-04T17:35:45.103 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:45.103 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:35:45.103 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:45.104 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-04T17:35:45.104 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-04T17:35:45.104 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-04T17:35:45.104 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-04T17:35:45.104 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-04T17:35:45.104 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:45.104 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:45.276 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 109 kB/s | 77 kB 00:00 2024-05-04T17:35:45.277 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:45.277 INFO:teuthology.orchestra.run.smithi186.stdout:Total 411 kB/s | 341 kB 00:00 2024-05-04T17:35:45.289 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:35:45.298 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:35:45.299 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:35:45.325 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:35:45.325 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:35:45.348 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-fuse 2024-05-04T17:35:45.466 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:35:45.660 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-04T17:35:45.718 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 2/2 2024-05-04T17:35:45.733 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 2/2 2024-05-04T17:35:45.734 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:35:45.734 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-04T17:35:45.734 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:45.886 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:39 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:46.012 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:46.013 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:46.013 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-05-04T17:35:46.013 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:46.013 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:35:46.013 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse x86_64 2:19.0.0-2497.gf273489c.el9 ceph 889 k 2024-05-04T17:35:46.013 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-05-04T17:35:46.013 INFO:teuthology.orchestra.run.smithi019.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-04T17:35:46.013 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:46.013 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:35:46.013 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:46.013 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-05-04T17:35:46.013 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:46.014 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 970 k 2024-05-04T17:35:46.014 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 2.8 M 2024-05-04T17:35:46.014 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:35:46.152 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 1.1 MB/s | 80 kB 00:00 2024-05-04T17:35:46.377 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 1/2 2024-05-04T17:35:46.535 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-fuse-19.0.0-2497.gf273489c.el9.x86_ 1.9 MB/s | 889 kB 00:00 2024-05-04T17:35:46.536 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:46.536 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.8 MB/s | 970 kB 00:00 2024-05-04T17:35:46.548 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:35:46.571 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:35:46.571 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:35:46.638 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-04T17:35:46.638 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:46.638 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:35:46.638 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:46.638 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-04T17:35:46.638 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:46.638 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:46.654 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:35:46.654 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:35:46.843 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:35:46.911 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados-devel 2024-05-04T17:35:46.983 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-05-04T17:35:47.053 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 2/2 2024-05-04T17:35:47.429 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:39 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:47.550 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:47.551 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:47.551 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:35:47.551 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:47.551 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:35:47.551 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-2497.gf273489c.el9 ceph 127 k 2024-05-04T17:35:47.551 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:47.552 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:35:47.552 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:47.552 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:35:47.552 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:47.552 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 127 k 2024-05-04T17:35:47.552 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 456 k 2024-05-04T17:35:47.552 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:35:47.667 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 2/2 2024-05-04T17:35:47.667 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 1/2 2024-05-04T17:35:47.863 INFO:teuthology.orchestra.run.smithi186.stdout:librados-devel-19.0.0-2497.gf273489c.el9.x86_64 407 kB/s | 127 kB 00:00 2024-05-04T17:35:47.864 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:47.864 INFO:teuthology.orchestra.run.smithi186.stdout:Total 405 kB/s | 127 kB 00:00 2024-05-04T17:35:47.864 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:35:47.870 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:35:47.870 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:35:47.920 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-05-04T17:35:47.920 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:47.920 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:35:47.920 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-05-04T17:35:47.920 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:47.920 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:47.934 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:35:47.934 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:35:48.088 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:35:48.207 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-volume 2024-05-04T17:35:48.211 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:48.574 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados-devel-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:48.750 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:42 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:48.830 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:48.830 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:48.830 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:35:48.830 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:48.830 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:48.830 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:48.882 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:48.883 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:48.883 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-05-04T17:35:48.883 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:48.883 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:35:48.883 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume noarch 2:19.0.0-2497.gf273489c.el9 ceph-noarch 264 k 2024-05-04T17:35:48.883 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-05-04T17:35:48.883 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-04T17:35:48.883 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:48.883 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:35:48.883 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:48.884 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-05-04T17:35:48.884 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:48.884 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 341 k 2024-05-04T17:35:48.884 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.5 M 2024-05-04T17:35:48.884 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:35:49.084 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs2 2024-05-04T17:35:49.149 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): ceph-volume-19.0.0-2497.gf273489c.el9.no 1.2 MB/s | 264 kB 00:00 2024-05-04T17:35:49.602 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:41 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:49.682 INFO:teuthology.orchestra.run.smithi186.stdout:Package libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:49.724 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:49.725 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-05-04T17:35:49.725 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:49.733 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 96 kB/s | 77 kB 00:00 2024-05-04T17:35:49.734 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:49.734 INFO:teuthology.orchestra.run.smithi019.stdout:Total 401 kB/s | 341 kB 00:00 2024-05-04T17:35:49.746 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:35:49.757 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:35:49.757 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:35:49.786 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:35:49.786 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:35:49.822 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs-devel 2024-05-04T17:35:49.935 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:35:50.124 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-05-04T17:35:50.152 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 2/2 2024-05-04T17:35:50.168 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 2/2 2024-05-04T17:35:50.168 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:35:50.168 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-04T17:35:50.168 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:50.342 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:42 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:50.463 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:50.463 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:50.463 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:35:50.464 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:50.464 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:35:50.464 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-2497.gf273489c.el9 ceph 32 k 2024-05-04T17:35:50.464 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:50.464 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:35:50.464 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:50.464 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:35:50.464 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:50.464 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 32 k 2024-05-04T17:35:50.464 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 142 k 2024-05-04T17:35:50.464 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:35:50.709 INFO:teuthology.orchestra.run.smithi186.stdout:libcephfs-devel-19.0.0-2497.gf273489c.el9.x86_6 130 kB/s | 32 kB 00:00 2024-05-04T17:35:50.710 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:50.710 INFO:teuthology.orchestra.run.smithi186.stdout:Total 129 kB/s | 32 kB 00:00 2024-05-04T17:35:50.710 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:35:50.713 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:35:50.714 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:35:50.716 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:35:50.716 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:35:50.794 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:35:50.806 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 1/2 2024-05-04T17:35:50.896 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:51.110 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-05-04T17:35:51.110 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:51.110 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:35:51.110 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume-2:19.0.0-2497.gf273489c.el9.noarch 2024-05-04T17:35:51.110 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-04T17:35:51.110 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:51.110 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:51.233 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:51.443 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:51.443 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:51.443 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:35:51.443 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:51.443 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:51.443 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:51.469 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados-devel 2024-05-04T17:35:51.626 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados2 2024-05-04T17:35:52.010 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:46 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:52.135 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:52.136 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:52.136 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:35:52.136 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:52.136 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:35:52.136 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel x86_64 2:19.0.0-2497.gf273489c.el9 ceph 127 k 2024-05-04T17:35:52.136 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:52.136 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:35:52.136 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:52.136 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:35:52.136 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:52.136 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 127 k 2024-05-04T17:35:52.137 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 456 k 2024-05-04T17:35:52.137 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:35:52.139 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:44 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:52.218 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:52.259 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:52.260 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-05-04T17:35:52.260 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:52.326 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librbd1 2024-05-04T17:35:52.367 INFO:teuthology.orchestra.run.smithi019.stdout:librados-devel-19.0.0-2497.gf273489c.el9.x86_64 552 kB/s | 127 kB 00:00 2024-05-04T17:35:52.367 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:52.367 INFO:teuthology.orchestra.run.smithi019.stdout:Total 548 kB/s | 127 kB 00:00 2024-05-04T17:35:52.367 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:35:52.373 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:35:52.373 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:35:52.441 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:35:52.441 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:35:52.601 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:35:52.707 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:52.841 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:44 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:52.920 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:52.961 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:52.963 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-05-04T17:35:52.963 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:53.022 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rados 2024-05-04T17:35:53.135 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados-devel-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:53.400 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:53.400 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:53.400 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:35:53.400 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:53.400 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:53.400 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:53.537 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:45 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:53.617 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:53.658 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:53.659 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-05-04T17:35:53.659 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:53.669 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs2 2024-05-04T17:35:53.723 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rgw 2024-05-04T17:35:54.216 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:48 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:54.233 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:46 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:54.297 INFO:teuthology.orchestra.run.smithi019.stdout:Package libcephfs2-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:54.311 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:54.341 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:54.343 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-05-04T17:35:54.343 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:54.352 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:54.353 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-05-04T17:35:54.353 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:54.402 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs-devel 2024-05-04T17:35:54.413 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-cephfs 2024-05-04T17:35:54.929 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:46 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:54.934 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:48 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:55.008 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:55.049 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:55.050 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-05-04T17:35:55.050 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:55.058 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:55.058 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:55.058 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:35:55.058 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:55.059 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:35:55.059 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel x86_64 2:19.0.0-2497.gf273489c.el9 ceph 32 k 2024-05-04T17:35:55.059 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:55.059 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:35:55.059 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:35:55.059 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:35:55.059 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:55.059 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 32 k 2024-05-04T17:35:55.060 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 142 k 2024-05-04T17:35:55.060 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:35:55.117 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rbd 2024-05-04T17:35:55.295 INFO:teuthology.orchestra.run.smithi019.stdout:libcephfs-devel-19.0.0-2497.gf273489c.el9.x86_6 135 kB/s | 32 kB 00:00 2024-05-04T17:35:55.295 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:55.295 INFO:teuthology.orchestra.run.smithi019.stdout:Total 134 kB/s | 32 kB 00:00 2024-05-04T17:35:55.296 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:35:55.299 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:35:55.299 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:35:55.301 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:35:55.302 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:35:55.394 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:35:55.510 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:55.634 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:47 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:55.713 INFO:teuthology.orchestra.run.smithi186.stdout:Package python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:55.755 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:55.756 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-05-04T17:35:55.756 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:55.818 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-fuse 2024-05-04T17:35:55.838 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:56.067 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:56.067 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:56.067 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:35:56.067 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:56.068 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:35:56.068 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:56.312 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados2 2024-05-04T17:35:56.333 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:48 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:56.454 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:56.455 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:56.455 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-05-04T17:35:56.455 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:56.455 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:35:56.455 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-2497.gf273489c.el9 ceph 85 k 2024-05-04T17:35:56.456 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:56.456 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:35:56.456 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:56.456 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:35:56.456 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:56.456 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 85 k 2024-05-04T17:35:56.456 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 231 k 2024-05-04T17:35:56.456 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:35:56.762 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-fuse-19.0.0-2497.gf273489c.el9.x86_64.rpm 279 kB/s | 85 kB 00:00 2024-05-04T17:35:56.763 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:56.763 INFO:teuthology.orchestra.run.smithi186.stdout:Total 278 kB/s | 85 kB 00:00 2024-05-04T17:35:56.763 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:35:56.770 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:35:56.770 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:35:56.833 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:35:56.833 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:35:56.852 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:50 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:56.936 INFO:teuthology.orchestra.run.smithi019.stdout:Package librados2-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:56.974 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:35:56.979 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:56.980 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-05-04T17:35:56.980 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:57.049 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librbd1 2024-05-04T17:35:57.084 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:57.429 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:57.587 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:51 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:57.660 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:57.660 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:57.660 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:35:57.660 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:35:57.660 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:57.661 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:35:57.670 INFO:teuthology.orchestra.run.smithi019.stdout:Package librbd1-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:57.714 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:57.715 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-05-04T17:35:57.715 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:57.776 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rados 2024-05-04T17:35:57.884 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-mirror 2024-05-04T17:35:58.319 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:52 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:58.400 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rados-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:58.434 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:50 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:35:58.444 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:58.446 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-05-04T17:35:58.446 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:58.507 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rgw 2024-05-04T17:35:58.558 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:35:58.559 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:58.559 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:35:58.559 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:58.559 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:35:58.559 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-2497.gf273489c.el9 ceph 3.1 M 2024-05-04T17:35:58.559 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:58.559 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:35:58.560 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:35:58.560 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:35:58.560 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:58.560 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 3.1 M 2024-05-04T17:35:58.560 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 13 M 2024-05-04T17:35:58.560 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:35:58.942 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-mirror-19.0.0-2497.gf273489c.el9.x86_64.rpm 8.1 MB/s | 3.1 MB 00:00 2024-05-04T17:35:58.943 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:35:58.943 INFO:teuthology.orchestra.run.smithi186.stdout:Total 8.0 MB/s | 3.1 MB 00:00 2024-05-04T17:35:58.943 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:35:58.952 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:35:58.953 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:35:59.033 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:35:59.034 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:35:59.049 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:53 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:59.132 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rgw-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:59.177 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:59.177 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-05-04T17:35:59.177 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:59.241 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-cephfs 2024-05-04T17:35:59.461 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:35:59.510 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:59.535 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:35:59.535 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:35:59.535 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-04T17:35:59.535 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-04T17:35:59.535 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-04T17:35:59.535 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:35:59.771 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:53 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:35:59.854 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-cephfs-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:35:59.897 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:35:59.898 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-05-04T17:35:59.899 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:35:59.963 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rbd 2024-05-04T17:36:00.504 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:54 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:36:00.585 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rbd-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:36:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:36:00.602 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:36:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:36:00.602 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:36:00.602 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:36:00.628 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:36:00.629 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-05-04T17:36:00.629 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:36:00.694 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-fuse 2024-05-04T17:36:00.811 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-nbd 2024-05-04T17:36:01.231 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:55 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:36:01.337 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:53 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:36:01.357 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:36:01.357 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:01.358 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-05-04T17:36:01.358 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:01.358 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:36:01.358 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse x86_64 2:19.0.0-2497.gf273489c.el9 ceph 85 k 2024-05-04T17:36:01.358 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:01.358 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:36:01.358 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:01.358 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:36:01.358 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:01.358 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 85 k 2024-05-04T17:36:01.358 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 231 k 2024-05-04T17:36:01.358 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:36:01.463 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:36:01.463 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:36:01.463 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-05-04T17:36:01.464 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:36:01.464 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:36:01.464 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-2497.gf273489c.el9 ceph 170 k 2024-05-04T17:36:01.464 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:36:01.464 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:36:01.464 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:36:01.464 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:36:01.464 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:36:01.465 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 170 k 2024-05-04T17:36:01.465 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 494 k 2024-05-04T17:36:01.465 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:36:01.698 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-fuse-19.0.0-2497.gf273489c.el9.x86_64.rpm 251 kB/s | 85 kB 00:00 2024-05-04T17:36:01.699 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:36:01.699 INFO:teuthology.orchestra.run.smithi019.stdout:Total 251 kB/s | 85 kB 00:00 2024-05-04T17:36:01.699 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:36:01.707 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:36:01.707 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:36:01.775 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:36:01.775 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:36:01.813 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-nbd-19.0.0-2497.gf273489c.el9.x86_64.rpm 490 kB/s | 170 kB 00:00 2024-05-04T17:36:01.813 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:36:01.813 INFO:teuthology.orchestra.run.smithi186.stdout:Total 488 kB/s | 170 kB 00:00 2024-05-04T17:36:01.813 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:36:01.822 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:36:01.822 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:36:01.886 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:36:01.886 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:36:01.930 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:36:02.030 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:02.033 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:36:02.125 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:02.393 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:02.523 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:02.660 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:36:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:36:02.660 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:02.660 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:36:02.767 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:02.767 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:36:02.768 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:36:02.768 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:36:02.768 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:36:02.768 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:36:02.857 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-mirror 2024-05-04T17:36:02.997 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-base-debuginfo 2024-05-04T17:36:03.399 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:57 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:36:03.518 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:01:55 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:36:03.526 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:36:03.526 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:03.527 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:36:03.527 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:03.527 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:36:03.527 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror x86_64 2:19.0.0-2497.gf273489c.el9 ceph 3.1 M 2024-05-04T17:36:03.527 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:03.527 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:36:03.527 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:03.527 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:36:03.527 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:03.528 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 3.1 M 2024-05-04T17:36:03.528 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 13 M 2024-05-04T17:36:03.528 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:36:03.641 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:36:03.641 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:36:03.641 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:36:03.641 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:36:03.642 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:36:03.642 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 69 M 2024-05-04T17:36:03.642 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-04T17:36:03.642 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 23 M 2024-05-04T17:36:03.642 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:36:03.642 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:36:03.642 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:36:03.642 INFO:teuthology.orchestra.run.smithi186.stdout:Install 2 Packages 2024-05-04T17:36:03.642 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:36:03.643 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 92 M 2024-05-04T17:36:03.643 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 503 M 2024-05-04T17:36:03.643 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:36:03.918 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-mirror-19.0.0-2497.gf273489c.el9.x86_64.rpm 7.9 MB/s | 3.1 MB 00:00 2024-05-04T17:36:03.918 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:36:03.919 INFO:teuthology.orchestra.run.smithi019.stdout:Total 7.9 MB/s | 3.1 MB 00:00 2024-05-04T17:36:03.919 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:36:03.929 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:36:03.929 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:36:04.015 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:36:04.015 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:36:04.470 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:36:04.661 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:04.688 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:04.688 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-04T17:36:04.688 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-04T17:36:04.688 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-04T17:36:04.688 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-04T17:36:04.688 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:04.975 INFO:teuthology.orchestra.run.smithi186.stdout:(1/2): ceph-debuginfo-19.0.0-2497.gf273489c.el9 18 MB/s | 23 MB 00:01 2024-05-04T17:36:05.667 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:05.667 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:05.667 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:36:05.667 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:36:05.667 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:05.667 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:36:05.883 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-nbd 2024-05-04T17:36:06.428 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:00 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:36:06.557 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:36:06.557 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:06.558 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-05-04T17:36:06.558 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:06.558 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:36:06.558 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd x86_64 2:19.0.0-2497.gf273489c.el9 ceph 170 k 2024-05-04T17:36:06.558 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:06.558 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:36:06.558 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:06.558 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:36:06.558 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:06.558 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 170 k 2024-05-04T17:36:06.558 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 494 k 2024-05-04T17:36:06.558 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:36:06.922 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-nbd-19.0.0-2497.gf273489c.el9.x86_64.rpm 469 kB/s | 170 kB 00:00 2024-05-04T17:36:06.922 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:36:06.923 INFO:teuthology.orchestra.run.smithi019.stdout:Total 467 kB/s | 170 kB 00:00 2024-05-04T17:36:06.923 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:36:06.931 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:36:06.931 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:36:06.998 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:36:06.999 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:36:07.162 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:36:07.271 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:07.665 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:07.908 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:36:07.908 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:07.908 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:36:07.908 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:36:07.908 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:07.908 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:36:08.124 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base-debuginfo 2024-05-04T17:36:08.566 INFO:teuthology.orchestra.run.smithi186.stdout:(2/2): ceph-base-debuginfo-19.0.0-2497.gf273489 14 MB/s | 69 MB 00:04 2024-05-04T17:36:08.568 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:36:08.568 INFO:teuthology.orchestra.run.smithi186.stdout:Total 19 MB/s | 92 MB 00:04 2024-05-04T17:36:08.569 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:36:08.573 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:36:08.573 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:36:08.664 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:02 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:36:08.790 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:36:08.791 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:36:08.791 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:08.791 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:36:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 69 M 2024-05-04T17:36:08.791 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-05-04T17:36:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 23 M 2024-05-04T17:36:08.791 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:08.791 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:36:08.791 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:08.792 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-05-04T17:36:08.792 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:08.792 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 92 M 2024-05-04T17:36:08.792 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 503 M 2024-05-04T17:36:08.792 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:36:09.161 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:36:09.161 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:36:11.816 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): ceph-debuginfo-19.0.0-2497.gf273489c.el9 7.7 MB/s | 23 MB 00:03 2024-05-04T17:36:12.360 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:36:13.250 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-base-debuginfo-19.0.0-2497.gf273489 15 MB/s | 69 MB 00:04 2024-05-04T17:36:13.252 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:36:13.252 INFO:teuthology.orchestra.run.smithi019.stdout:Total 21 MB/s | 92 MB 00:04 2024-05-04T17:36:13.252 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:36:13.257 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:36:13.258 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:36:13.866 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:36:13.867 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:36:17.233 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:36:19.500 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/2 2024-05-04T17:36:19.574 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 2/2 2024-05-04T17:36:22.920 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 2/2 2024-05-04T17:36:22.921 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/2 2024-05-04T17:36:23.138 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2/2 2024-05-04T17:36:23.138 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:36:23.139 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:36:23.139 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:36:23.139 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:36:23.139 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:36:23.139 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:36:23.353 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-common-debuginfo 2024-05-04T17:36:23.862 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:02:15 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:36:23.982 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:36:23.983 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:36:23.983 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:36:23.984 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:36:23.984 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:36:23.984 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 451 M 2024-05-04T17:36:23.984 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:36:23.984 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:36:23.984 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:36:23.984 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:36:23.984 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:36:23.984 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 451 M 2024-05-04T17:36:23.984 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 1.8 G 2024-05-04T17:36:23.984 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:36:24.786 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/2 2024-05-04T17:36:24.860 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 2/2 2024-05-04T17:36:28.155 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 2/2 2024-05-04T17:36:28.155 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/2 2024-05-04T17:36:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2/2 2024-05-04T17:36:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:28.389 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:36:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:36:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:36:28.389 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:28.389 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:36:28.630 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-common-debuginfo 2024-05-04T17:36:29.175 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:23 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:36:29.306 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:36:29.306 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:29.307 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:36:29.307 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:29.307 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:36:29.307 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 451 M 2024-05-04T17:36:29.307 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:29.307 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:36:29.307 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:36:29.307 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:36:29.307 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:36:29.307 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 451 M 2024-05-04T17:36:29.307 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.8 G 2024-05-04T17:36:29.307 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:36:47.621 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-common-debuginfo-19.0.0-2497.gf273489c.el9 19 MB/s | 451 MB 00:23 2024-05-04T17:36:47.622 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:36:47.622 INFO:teuthology.orchestra.run.smithi186.stdout:Total 19 MB/s | 451 MB 00:23 2024-05-04T17:36:47.623 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:36:47.627 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:36:47.627 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:36:50.299 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:36:50.299 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:37:06.319 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-common-debuginfo-19.0.0-2497.gf273489c.el9 12 MB/s | 451 MB 00:37 2024-05-04T17:37:06.320 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:37:06.320 INFO:teuthology.orchestra.run.smithi019.stdout:Total 12 MB/s | 451 MB 00:37 2024-05-04T17:37:06.321 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:37:06.325 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:37:06.325 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:37:09.133 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:37:09.134 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:37:35.987 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:37:36.165 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:37:44.533 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:37:44.742 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:37:44.742 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:37:44.742 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:37:44.742 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:37:44.742 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:37:44.743 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:37:44.939 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-05-04T17:37:45.454 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:03:37 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:37:45.577 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:37:45.578 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-05-04T17:37:45.578 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:37:45.578 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-05-04T17:37:45.578 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:37:45.578 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 1.5 M 2024-05-04T17:37:45.578 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:37:45.579 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:37:45.579 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================== 2024-05-04T17:37:45.579 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:37:45.579 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:37:45.579 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 1.5 M 2024-05-04T17:37:45.579 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 6.6 M 2024-05-04T17:37:45.579 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:37:46.061 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-24 3.0 MB/s | 1.5 MB 00:00 2024-05-04T17:37:46.062 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:37:46.063 INFO:teuthology.orchestra.run.smithi186.stdout:Total 3.0 MB/s | 1.5 MB 00:00 2024-05-04T17:37:46.063 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:37:46.066 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:37:46.066 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:37:46.109 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:37:46.109 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:37:46.367 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:37:46.473 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-2497. 1/1 2024-05-04T17:37:46.802 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-2497. 1/1 2024-05-04T17:37:47.001 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-2497. 1/1 2024-05-04T17:37:47.002 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:37:47.002 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:37:47.002 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:37:47.002 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:37:47.002 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:37:47.162 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw-debuginfo 2024-05-04T17:37:47.676 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:03:39 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:37:47.797 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:37:47.798 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:37:47.798 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:37:47.798 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:37:47.798 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:37:47.798 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 187 M 2024-05-04T17:37:47.798 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:37:47.798 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:37:47.798 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:37:47.798 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:37:47.798 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:37:47.798 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 187 M 2024-05-04T17:37:47.799 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 796 M 2024-05-04T17:37:47.799 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:37:57.596 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:37:57.699 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:37:57.747 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-radosgw-debuginfo-19.0.0-2497.gf273489c.el 19 MB/s | 187 MB 00:09 2024-05-04T17:37:57.749 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:37:57.749 INFO:teuthology.orchestra.run.smithi186.stdout:Total 19 MB/s | 187 MB 00:09 2024-05-04T17:37:57.749 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:37:57.753 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:37:57.753 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:37:58.877 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:37:58.878 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:38:05.095 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:38:05.319 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:38:05.319 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:38:05.319 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:38:05.319 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:38:05.319 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:38:05.320 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:38:05.567 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-05-04T17:38:06.114 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:04:00 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:38:06.242 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:38:06.243 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-05-04T17:38:06.243 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:38:06.243 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-05-04T17:38:06.243 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:38:06.243 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 1.5 M 2024-05-04T17:38:06.243 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:38:06.243 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:38:06.243 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-05-04T17:38:06.243 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:38:06.243 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:38:06.244 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 1.5 M 2024-05-04T17:38:06.244 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 6.6 M 2024-05-04T17:38:06.244 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:38:06.719 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-24 3.1 MB/s | 1.5 MB 00:00 2024-05-04T17:38:06.719 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:38:06.720 INFO:teuthology.orchestra.run.smithi019.stdout:Total 3.1 MB/s | 1.5 MB 00:00 2024-05-04T17:38:06.720 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:38:06.724 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:38:06.724 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:38:06.770 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:38:06.770 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:38:07.041 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:38:07.142 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-2497. 1/1 2024-05-04T17:38:07.468 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-2497. 1/1 2024-05-04T17:38:07.693 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-2497. 1/1 2024-05-04T17:38:07.693 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:38:07.693 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:38:07.693 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:38:07.693 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:38:07.693 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:38:07.858 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw-debuginfo 2024-05-04T17:38:08.387 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:04:02 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:38:08.513 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:38:08.514 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:38:08.514 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:38:08.514 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:38:08.514 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:38:08.514 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 187 M 2024-05-04T17:38:08.514 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:38:08.514 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:38:08.514 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:38:08.514 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:38:08.515 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:38:08.515 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 187 M 2024-05-04T17:38:08.515 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 796 M 2024-05-04T17:38:08.515 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:38:16.779 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-radosgw-debuginfo-19.0.0-2497.gf273489c.el 23 MB/s | 187 MB 00:08 2024-05-04T17:38:16.780 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:38:16.780 INFO:teuthology.orchestra.run.smithi019.stdout:Total 23 MB/s | 187 MB 00:08 2024-05-04T17:38:16.781 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:38:16.785 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:38:16.785 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:38:17.971 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:38:17.972 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:38:18.268 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:38:18.366 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x 1/1 2024-05-04T17:38:23.126 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x 1/1 2024-05-04T17:38:23.337 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x 1/1 2024-05-04T17:38:23.337 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:38:23.337 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:38:23.337 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:38:23.337 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:38:23.337 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:38:23.519 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-test-debuginfo 2024-05-04T17:38:24.049 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:04:16 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:38:24.179 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:38:24.179 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:38:24.179 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:38:24.180 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:38:24.180 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:38:24.180 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 692 M 2024-05-04T17:38:24.180 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:38:24.180 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:38:24.180 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:38:24.180 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:38:24.180 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:38:24.181 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 692 M 2024-05-04T17:38:24.181 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 3.2 G 2024-05-04T17:38:24.181 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:38:38.283 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:38:38.368 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x 1/1 2024-05-04T17:38:43.348 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x 1/1 2024-05-04T17:38:43.579 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x 1/1 2024-05-04T17:38:43.580 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:38:43.580 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:38:43.580 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:38:43.580 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:38:43.580 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:38:43.803 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-test-debuginfo 2024-05-04T17:38:44.342 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:04:38 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:38:44.468 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:38:44.468 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:38:44.468 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:38:44.468 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:38:44.469 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:38:44.469 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 692 M 2024-05-04T17:38:44.469 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:38:44.469 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:38:44.469 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:38:44.469 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:38:44.469 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:38:44.469 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 692 M 2024-05-04T17:38:44.469 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 3.2 G 2024-05-04T17:38:44.469 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:38:57.437 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-test-debuginfo-19.0.0-2497.gf273489c.el9.x 21 MB/s | 692 MB 00:33 2024-05-04T17:38:57.438 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:38:57.439 INFO:teuthology.orchestra.run.smithi186.stdout:Total 21 MB/s | 692 MB 00:33 2024-05-04T17:38:57.439 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:38:57.445 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:38:57.445 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:39:01.523 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:39:01.523 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:39:19.720 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-test-debuginfo-19.0.0-2497.gf273489c.el9.x 20 MB/s | 692 MB 00:35 2024-05-04T17:39:19.720 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:39:19.721 INFO:teuthology.orchestra.run.smithi019.stdout:Total 20 MB/s | 692 MB 00:35 2024-05-04T17:39:19.721 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:39:19.727 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:39:19.727 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:39:24.012 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:39:24.012 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:40:14.545 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:40:14.660 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:40:23.911 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:40:24.120 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:40:24.120 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:40:24.120 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:40:24.120 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:40:24.120 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:40:24.120 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:40:24.389 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-base-debuginfo 2024-05-04T17:40:24.904 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:06:16 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:40:24.982 INFO:teuthology.orchestra.run.smithi186.stdout:Package ceph-base-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:40:25.024 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:40:25.025 INFO:teuthology.orchestra.run.smithi186.stdout:Nothing to do. 2024-05-04T17:40:25.025 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:40:25.090 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mgr-debuginfo 2024-05-04T17:40:25.603 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:06:17 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:40:25.723 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:40:25.723 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:40:25.724 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:40:25.724 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:40:25.724 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:40:25.724 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 20 M 2024-05-04T17:40:25.724 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:40:25.724 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:40:25.724 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:40:25.724 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:40:25.724 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:40:25.724 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 20 M 2024-05-04T17:40:25.725 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 104 M 2024-05-04T17:40:25.725 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:40:26.956 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-mgr-debuginfo-19.0.0-2497.gf273489c.el9.x8 16 MB/s | 20 MB 00:01 2024-05-04T17:40:26.956 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:40:26.957 INFO:teuthology.orchestra.run.smithi186.stdout:Total 16 MB/s | 20 MB 00:01 2024-05-04T17:40:26.957 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:40:26.960 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:40:26.961 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:40:27.113 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:40:27.113 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:40:29.370 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:40:29.543 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:40:30.426 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:40:30.636 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:40:30.636 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:40:30.636 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:40:30.636 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:40:30.636 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:40:30.636 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:40:30.805 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mds-debuginfo 2024-05-04T17:40:31.309 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:06:23 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:40:31.430 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:40:31.430 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:40:31.430 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:40:31.430 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:40:31.430 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:40:31.430 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 53 M 2024-05-04T17:40:31.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:40:31.431 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:40:31.431 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:40:31.431 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:40:31.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:40:31.431 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 53 M 2024-05-04T17:40:31.431 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 195 M 2024-05-04T17:40:31.431 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:40:34.783 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-mds-debuginfo-19.0.0-2497.gf273489c.el9.x8 16 MB/s | 53 MB 00:03 2024-05-04T17:40:34.784 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:40:34.784 INFO:teuthology.orchestra.run.smithi186.stdout:Total 16 MB/s | 53 MB 00:03 2024-05-04T17:40:34.785 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:40:34.789 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:40:34.789 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:40:35.135 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:40:35.136 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:40:40.446 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:40:40.549 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:40:41.197 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:40:41.303 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:40:42.001 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:40:42.226 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:40:42.226 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:40:42.226 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:40:42.226 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:40:42.226 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:40:42.226 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:40:42.379 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-mon-debuginfo 2024-05-04T17:40:42.894 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:06:34 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:40:43.014 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:40:43.015 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:40:43.015 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:40:43.015 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:40:43.015 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:40:43.015 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 99 M 2024-05-04T17:40:43.015 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:40:43.015 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:40:43.015 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:40:43.015 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:40:43.015 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:40:43.016 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 99 M 2024-05-04T17:40:43.016 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 450 M 2024-05-04T17:40:43.016 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:40:49.106 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:40:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:40:49.329 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:40:49.330 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:40:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:40:49.330 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:40:49.330 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:40:49.634 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base-debuginfo 2024-05-04T17:40:50.168 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:44 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:40:50.244 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-mon-debuginfo-19.0.0-2497.gf273489c.el9.x8 14 MB/s | 99 MB 00:07 2024-05-04T17:40:50.245 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:40:50.246 INFO:teuthology.orchestra.run.smithi186.stdout:Total 14 MB/s | 99 MB 00:07 2024-05-04T17:40:50.246 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:40:50.250 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:40:50.250 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:40:50.251 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-base-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 is already installed. 2024-05-04T17:40:50.294 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:40:50.295 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-05-04T17:40:50.296 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:40:50.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-debuginfo 2024-05-04T17:40:50.862 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:40:50.862 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:40:50.903 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:44 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:40:51.029 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:40:51.030 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:40:51.030 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:40:51.030 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:40:51.030 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:40:51.030 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 20 M 2024-05-04T17:40:51.030 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:40:51.030 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:40:51.030 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:40:51.030 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:40:51.030 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:40:51.030 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 20 M 2024-05-04T17:40:51.031 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 104 M 2024-05-04T17:40:51.031 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:40:53.682 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-mgr-debuginfo-19.0.0-2497.gf273489c.el9.x8 7.6 MB/s | 20 MB 00:02 2024-05-04T17:40:53.683 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:40:53.684 INFO:teuthology.orchestra.run.smithi019.stdout:Total 7.6 MB/s | 20 MB 00:02 2024-05-04T17:40:53.684 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:40:53.688 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:40:53.688 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:40:53.850 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:40:53.850 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:40:56.254 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:40:56.356 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:40:57.416 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:40:57.653 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:40:57.653 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:40:57.654 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:40:57.654 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:40:57.654 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:40:57.654 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:40:57.810 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mds-debuginfo 2024-05-04T17:40:58.345 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:06:52 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:40:58.471 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:40:58.472 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:40:58.472 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:40:58.473 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:40:58.473 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:40:58.473 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 53 M 2024-05-04T17:40:58.473 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:40:58.473 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:40:58.473 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:40:58.473 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:40:58.473 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:40:58.473 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 53 M 2024-05-04T17:40:58.473 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 195 M 2024-05-04T17:40:58.473 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:41:01.146 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:41:01.247 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:41:01.379 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-mds-debuginfo-19.0.0-2497.gf273489c.el9.x8 18 MB/s | 53 MB 00:02 2024-05-04T17:41:01.380 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:41:01.380 INFO:teuthology.orchestra.run.smithi019.stdout:Total 18 MB/s | 53 MB 00:02 2024-05-04T17:41:01.381 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:41:01.385 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:41:01.385 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:41:01.751 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:41:01.751 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:41:04.074 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:41:04.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:41:04.281 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:41:04.281 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:41:04.281 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:41:04.281 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:41:04.281 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:41:04.456 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-osd-debuginfo 2024-05-04T17:41:04.967 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:06:56 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:41:05.087 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:41:05.087 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:41:05.088 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:41:05.088 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:41:05.088 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:41:05.088 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 288 M 2024-05-04T17:41:05.088 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:41:05.088 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:41:05.088 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:41:05.088 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:41:05.088 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:41:05.088 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 288 M 2024-05-04T17:41:05.088 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 1.2 G 2024-05-04T17:41:05.088 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:41:07.390 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:41:07.487 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:41:08.923 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:41:09.151 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:41:09.152 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:41:09.152 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:41:09.152 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:41:09.152 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:41:09.152 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:41:09.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mon-debuginfo 2024-05-04T17:41:09.858 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:07:03 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:41:09.983 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 99 M 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 99 M 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 450 M 2024-05-04T17:41:09.984 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:41:19.537 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-osd-debuginfo-19.0.0-2497.gf273489c.el9.x8 20 MB/s | 288 MB 00:14 2024-05-04T17:41:19.537 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:41:19.538 INFO:teuthology.orchestra.run.smithi186.stdout:Total 20 MB/s | 288 MB 00:14 2024-05-04T17:41:19.538 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:41:19.542 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:41:19.542 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:41:21.259 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:41:21.259 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:41:22.431 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-mon-debuginfo-19.0.0-2497.gf273489c.el9.x8 7.9 MB/s | 99 MB 00:12 2024-05-04T17:41:22.432 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:41:22.432 INFO:teuthology.orchestra.run.smithi019.stdout:Total 7.9 MB/s | 99 MB 00:12 2024-05-04T17:41:22.433 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:41:22.436 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:41:22.436 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:41:23.082 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:41:23.082 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:41:34.026 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:41:34.121 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:41:36.940 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:41:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:41:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:41:37.164 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:41:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:41:37.164 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:41:37.164 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:41:37.346 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-osd-debuginfo 2024-05-04T17:41:37.883 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:07:31 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:41:38.008 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:41:38.009 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:41:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:41:38.009 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:41:38.009 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:41:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 288 M 2024-05-04T17:41:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:41:38.009 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:41:38.009 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:41:38.009 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:41:38.009 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:41:38.009 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 288 M 2024-05-04T17:41:38.010 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.2 G 2024-05-04T17:41:38.010 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:41:50.590 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-osd-debuginfo-19.0.0-2497.gf273489c.el9.x8 23 MB/s | 288 MB 00:12 2024-05-04T17:41:50.591 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:41:50.591 INFO:teuthology.orchestra.run.smithi019.stdout:Total 23 MB/s | 288 MB 00:12 2024-05-04T17:41:50.592 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:41:50.596 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:41:50.596 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:41:50.844 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:41:50.972 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:41:52.401 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:41:52.402 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:41:58.387 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:41:58.605 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:41:58.605 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:41:58.605 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:41:58.605 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:41:58.605 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:41:58.605 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:41:58.789 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-fuse-debuginfo 2024-05-04T17:41:59.299 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:07:51 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:41:59.419 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:41:59.419 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:41:59.420 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:41:59.420 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:41:59.420 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:41:59.420 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 6.0 M 2024-05-04T17:41:59.420 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:41:59.420 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:41:59.420 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:41:59.420 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:41:59.420 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:41:59.420 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 6.0 M 2024-05-04T17:41:59.420 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 28 M 2024-05-04T17:41:59.420 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:42:00.563 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-fuse-debuginfo-19.0.0-2497.gf273489c.el9.x 5.3 MB/s | 6.0 MB 00:01 2024-05-04T17:42:00.564 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:00.564 INFO:teuthology.orchestra.run.smithi186.stdout:Total 5.2 MB/s | 6.0 MB 00:01 2024-05-04T17:42:00.565 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:42:00.569 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:42:00.569 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:42:00.642 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:42:00.643 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:42:01.395 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:42:01.501 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:42:02.002 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:42:02.205 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:42:02.205 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:02.205 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:42:02.205 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:02.205 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:02.205 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:42:02.362 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados-devel-debuginfo 2024-05-04T17:42:02.879 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:07:54 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:42:03.000 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:42:03.001 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:03.001 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:42:03.001 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:03.002 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:42:03.002 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 109 k 2024-05-04T17:42:03.002 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:03.002 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:42:03.002 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:03.002 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:42:03.002 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:03.002 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 109 k 2024-05-04T17:42:03.002 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 440 k 2024-05-04T17:42:03.002 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:42:03.219 INFO:teuthology.orchestra.run.smithi186.stdout:librados-devel-debuginfo-19.0.0-2497.gf273489c. 501 kB/s | 109 kB 00:00 2024-05-04T17:42:03.220 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:03.220 INFO:teuthology.orchestra.run.smithi186.stdout:Total 498 kB/s | 109 kB 00:00 2024-05-04T17:42:03.220 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:42:03.223 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:42:03.223 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:42:03.258 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:42:03.259 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:42:03.374 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:42:03.482 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-debuginfo-2:19.0.0-2497.gf273489c.el9 1/1 2024-05-04T17:42:03.811 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-2497.gf273489c.el9 1/1 2024-05-04T17:42:04.018 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-2497.gf273489c.el9 1/1 2024-05-04T17:42:04.018 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:04.018 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:42:04.018 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:04.018 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:04.018 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:42:04.167 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install libcephfs2-debuginfo 2024-05-04T17:42:04.677 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:07:56 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:42:04.798 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:42:04.798 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:04.799 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:42:04.799 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:04.799 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:42:04.799 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 5.4 M 2024-05-04T17:42:04.799 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:04.799 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:42:04.799 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:04.799 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:42:04.799 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:04.799 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 5.4 M 2024-05-04T17:42:04.799 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 25 M 2024-05-04T17:42:04.799 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:42:05.299 INFO:teuthology.orchestra.run.smithi186.stdout:libcephfs2-debuginfo-19.0.0-2497.gf273489c.el9. 11 MB/s | 5.4 MB 00:00 2024-05-04T17:42:05.300 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:05.300 INFO:teuthology.orchestra.run.smithi186.stdout:Total 11 MB/s | 5.4 MB 00:00 2024-05-04T17:42:05.301 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:42:05.304 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:42:05.304 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:42:05.371 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:42:05.371 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:42:06.044 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:42:06.164 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86 1/1 2024-05-04T17:42:06.606 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86 1/1 2024-05-04T17:42:06.825 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86 1/1 2024-05-04T17:42:06.825 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:06.825 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:42:06.825 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:06.825 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:06.825 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:42:07.002 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librados2-debuginfo 2024-05-04T17:42:07.520 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:07:59 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:42:07.643 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:42:07.643 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:07.644 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:42:07.644 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:07.644 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:42:07.644 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 45 M 2024-05-04T17:42:07.644 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:07.644 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:42:07.644 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:07.644 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:42:07.644 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:07.644 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 45 M 2024-05-04T17:42:07.644 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 256 M 2024-05-04T17:42:07.644 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:42:10.355 INFO:teuthology.orchestra.run.smithi186.stdout:librados2-debuginfo-19.0.0-2497.gf273489c.el9.x 17 MB/s | 45 MB 00:02 2024-05-04T17:42:10.356 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:10.356 INFO:teuthology.orchestra.run.smithi186.stdout:Total 17 MB/s | 45 MB 00:02 2024-05-04T17:42:10.356 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:42:10.361 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:42:10.361 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:42:10.664 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:42:10.664 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:42:15.802 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:42:15.883 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:42:17.711 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:42:17.932 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:42:17.932 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:17.933 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:42:17.933 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:17.933 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:17.933 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:42:18.111 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install librbd1-debuginfo 2024-05-04T17:42:18.623 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:08:10 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:42:18.746 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:42:18.747 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:18.747 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:42:18.747 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:18.747 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:42:18.747 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 52 M 2024-05-04T17:42:18.747 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:18.747 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:42:18.747 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:18.747 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:42:18.748 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:18.748 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 52 M 2024-05-04T17:42:18.748 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 308 M 2024-05-04T17:42:18.748 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:42:22.333 INFO:teuthology.orchestra.run.smithi186.stdout:librbd1-debuginfo-19.0.0-2497.gf273489c.el9.x86 15 MB/s | 52 MB 00:03 2024-05-04T17:42:22.334 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:22.334 INFO:teuthology.orchestra.run.smithi186.stdout:Total 14 MB/s | 52 MB 00:03 2024-05-04T17:42:22.335 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:42:22.339 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:42:22.339 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:42:22.685 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:42:22.685 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:42:23.700 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:42:23.796 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:42:28.740 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:42:28.872 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librbd1-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:42:30.977 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:42:31.191 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:42:31.192 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:31.192 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:42:31.192 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:31.192 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:31.192 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:42:31.245 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:42:31.355 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-cephfs-debuginfo 2024-05-04T17:42:31.486 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:42:31.486 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:31.486 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:42:31.486 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:31.486 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:31.486 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:42:31.699 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-fuse-debuginfo 2024-05-04T17:42:31.875 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:08:23 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:42:31.996 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:42:31.997 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:31.997 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:42:31.997 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:31.997 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:42:31.997 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 615 k 2024-05-04T17:42:31.997 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:31.997 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:42:31.998 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:31.998 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:42:31.998 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:31.998 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 615 k 2024-05-04T17:42:31.998 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 2.1 M 2024-05-04T17:42:31.998 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:42:32.237 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:08:26 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:42:32.362 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:42:32.363 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:32.363 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:42:32.363 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:32.364 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:42:32.364 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 6.0 M 2024-05-04T17:42:32.364 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:32.364 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:42:32.364 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:32.365 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:42:32.365 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:32.365 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 6.0 M 2024-05-04T17:42:32.365 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 28 M 2024-05-04T17:42:32.365 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:42:32.428 INFO:teuthology.orchestra.run.smithi186.stdout:python3-cephfs-debuginfo-19.0.0-2497.gf273489c. 1.4 MB/s | 615 kB 00:00 2024-05-04T17:42:32.428 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:32.428 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.4 MB/s | 615 kB 00:00 2024-05-04T17:42:32.429 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:42:32.432 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:42:32.432 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:42:32.471 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:42:32.472 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:42:32.638 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:42:32.742 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-2497.gf273489c.el9 1/1 2024-05-04T17:42:32.899 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-fuse-debuginfo-19.0.0-2497.gf273489c.el9.x 11 MB/s | 6.0 MB 00:00 2024-05-04T17:42:32.899 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:32.900 INFO:teuthology.orchestra.run.smithi019.stdout:Total 11 MB/s | 6.0 MB 00:00 2024-05-04T17:42:32.900 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:42:32.904 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:42:32.904 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:42:32.981 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:42:32.981 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:42:33.111 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-2497.gf273489c.el9 1/1 2024-05-04T17:42:33.326 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-2497.gf273489c.el9 1/1 2024-05-04T17:42:33.326 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:33.326 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:42:33.326 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:33.326 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:33.326 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:42:33.485 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rados-debuginfo 2024-05-04T17:42:33.762 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:42:33.848 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:42:33.998 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:08:25 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:42:34.118 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:42:34.119 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:34.119 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:42:34.119 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:34.119 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:42:34.119 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 770 k 2024-05-04T17:42:34.119 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:34.119 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:42:34.120 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:34.120 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:42:34.120 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:34.120 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 770 k 2024-05-04T17:42:34.120 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 2.7 M 2024-05-04T17:42:34.120 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:42:34.376 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:42:34.422 INFO:teuthology.orchestra.run.smithi186.stdout:python3-rados-debuginfo-19.0.0-2497.gf273489c.e 2.5 MB/s | 770 kB 00:00 2024-05-04T17:42:34.423 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:34.423 INFO:teuthology.orchestra.run.smithi186.stdout:Total 2.5 MB/s | 770 kB 00:00 2024-05-04T17:42:34.423 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:42:34.427 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:42:34.427 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:42:34.466 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:42:34.466 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:42:34.596 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:42:34.596 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:34.596 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:42:34.596 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:34.596 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:34.596 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:42:34.660 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:42:34.748 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados-devel-debuginfo 2024-05-04T17:42:34.760 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-debuginfo-2:19.0.0-2497.gf273489c.el9. 1/1 2024-05-04T17:42:35.074 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-2497.gf273489c.el9. 1/1 2024-05-04T17:42:35.282 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:08:29 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:42:35.304 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-2497.gf273489c.el9. 1/1 2024-05-04T17:42:35.304 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:35.304 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:42:35.304 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:35.304 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:35.305 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:42:35.408 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:42:35.409 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:35.409 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:42:35.409 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:35.409 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:42:35.409 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 109 k 2024-05-04T17:42:35.409 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:35.409 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:42:35.409 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:35.409 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:42:35.410 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:35.410 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 109 k 2024-05-04T17:42:35.410 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 440 k 2024-05-04T17:42:35.410 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:42:35.461 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rbd-debuginfo 2024-05-04T17:42:35.627 INFO:teuthology.orchestra.run.smithi019.stdout:librados-devel-debuginfo-19.0.0-2497.gf273489c. 502 kB/s | 109 kB 00:00 2024-05-04T17:42:35.628 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:35.628 INFO:teuthology.orchestra.run.smithi019.stdout:Total 499 kB/s | 109 kB 00:00 2024-05-04T17:42:35.628 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:42:35.631 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:42:35.631 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:42:35.670 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:42:35.670 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:42:35.794 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:42:35.905 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-debuginfo-2:19.0.0-2497.gf273489c.el9 1/1 2024-05-04T17:42:35.969 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:08:27 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:42:36.089 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:42:36.090 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:36.090 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:42:36.090 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:36.090 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:42:36.090 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 1.1 M 2024-05-04T17:42:36.090 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:36.090 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:42:36.091 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:36.091 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:42:36.091 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:36.091 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 1.1 M 2024-05-04T17:42:36.091 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 4.3 M 2024-05-04T17:42:36.091 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:42:36.206 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-2497.gf273489c.el9 1/1 2024-05-04T17:42:36.515 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-2497.gf273489c.el9 1/1 2024-05-04T17:42:36.516 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:36.516 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:42:36.516 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:36.516 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:36.516 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:42:36.559 INFO:teuthology.orchestra.run.smithi186.stdout:python3-rbd-debuginfo-19.0.0-2497.gf273489c.el9 2.4 MB/s | 1.1 MB 00:00 2024-05-04T17:42:36.559 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:36.560 INFO:teuthology.orchestra.run.smithi186.stdout:Total 2.4 MB/s | 1.1 MB 00:00 2024-05-04T17:42:36.560 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:42:36.563 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:42:36.563 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:42:36.604 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:42:36.605 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:42:36.693 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs2-debuginfo 2024-05-04T17:42:36.822 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:42:36.904 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:42:37.224 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:08:31 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:42:37.244 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:42:37.350 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:42:37.350 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:37.350 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:42:37.351 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:37.351 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:42:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 5.4 M 2024-05-04T17:42:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:37.351 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:42:37.351 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:37.351 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:42:37.351 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:37.351 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 5.4 M 2024-05-04T17:42:37.351 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 25 M 2024-05-04T17:42:37.351 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:42:37.447 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:42:37.447 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:37.447 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:42:37.447 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:37.447 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:37.447 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:42:37.599 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install python3-rgw-debuginfo 2024-05-04T17:42:37.873 INFO:teuthology.orchestra.run.smithi019.stdout:libcephfs2-debuginfo-19.0.0-2497.gf273489c.el9. 10 MB/s | 5.4 MB 00:00 2024-05-04T17:42:37.874 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:37.874 INFO:teuthology.orchestra.run.smithi019.stdout:Total 10 MB/s | 5.4 MB 00:00 2024-05-04T17:42:37.874 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:42:37.878 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:42:37.878 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:42:37.949 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:42:37.949 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:42:38.105 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:08:30 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:42:38.228 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:42:38.229 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:38.229 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:42:38.229 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:38.230 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:42:38.230 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 337 k 2024-05-04T17:42:38.230 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:38.230 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:42:38.230 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:38.230 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:42:38.230 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:38.230 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 337 k 2024-05-04T17:42:38.230 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 1.0 M 2024-05-04T17:42:38.230 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:42:38.651 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:42:38.653 INFO:teuthology.orchestra.run.smithi186.stdout:python3-rgw-debuginfo-19.0.0-2497.gf273489c.el9 797 kB/s | 337 kB 00:00 2024-05-04T17:42:38.654 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:38.655 INFO:teuthology.orchestra.run.smithi186.stdout:Total 794 kB/s | 337 kB 00:00 2024-05-04T17:42:38.655 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:42:38.658 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:42:38.658 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:42:38.695 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:42:38.696 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:42:38.756 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86 1/1 2024-05-04T17:42:38.834 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:42:38.933 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:42:39.229 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86 1/1 2024-05-04T17:42:39.240 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:42:39.454 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:42:39.454 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:39.454 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:42:39.454 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:39.454 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:39.454 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:42:39.464 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86 1/1 2024-05-04T17:42:39.464 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:39.464 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:42:39.464 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:39.464 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:39.464 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:42:39.626 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-fuse-debuginfo 2024-05-04T17:42:39.664 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados2-debuginfo 2024-05-04T17:42:40.135 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:08:32 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:42:40.201 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:08:34 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:42:40.256 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:42:40.257 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:40.257 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:42:40.257 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:40.257 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:42:40.257 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 516 k 2024-05-04T17:42:40.257 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:40.257 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:42:40.257 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:40.257 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:42:40.258 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:40.258 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 516 k 2024-05-04T17:42:40.258 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 2.6 M 2024-05-04T17:42:40.258 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:42:40.328 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:42:40.328 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:40.328 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:42:40.328 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:40.328 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:42:40.328 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 45 M 2024-05-04T17:42:40.328 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:40.329 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:42:40.329 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:40.329 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:42:40.329 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:40.330 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 45 M 2024-05-04T17:42:40.330 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 256 M 2024-05-04T17:42:40.330 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:42:40.748 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-fuse-debuginfo-19.0.0-2497.gf273489c.el9.x8 1.0 MB/s | 516 kB 00:00 2024-05-04T17:42:40.749 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:40.749 INFO:teuthology.orchestra.run.smithi186.stdout:Total 1.0 MB/s | 516 kB 00:00 2024-05-04T17:42:40.749 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:42:40.753 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:42:40.753 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:42:40.791 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:42:40.792 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:42:40.949 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:42:41.036 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:42:41.371 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:42:41.579 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:42:41.579 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:41.579 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:42:41.579 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:41.579 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:41.579 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:42:41.764 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-mirror-debuginfo 2024-05-04T17:42:42.274 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:08:34 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:42:42.395 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:42:42.396 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:42.396 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:42:42.396 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:42.396 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:42:42.396 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 52 M 2024-05-04T17:42:42.396 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:42.396 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:42:42.396 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:42.397 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:42:42.397 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:42.397 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 52 M 2024-05-04T17:42:42.397 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 298 M 2024-05-04T17:42:42.397 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:42:42.828 INFO:teuthology.orchestra.run.smithi019.stdout:librados2-debuginfo-19.0.0-2497.gf273489c.el9.x 18 MB/s | 45 MB 00:02 2024-05-04T17:42:42.829 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:42.829 INFO:teuthology.orchestra.run.smithi019.stdout:Total 18 MB/s | 45 MB 00:02 2024-05-04T17:42:42.830 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:42:42.834 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:42:42.835 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:42:43.153 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:42:43.153 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:42:46.151 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-mirror-debuginfo-19.0.0-2497.gf273489c.el9. 14 MB/s | 52 MB 00:03 2024-05-04T17:42:46.152 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:46.153 INFO:teuthology.orchestra.run.smithi186.stdout:Total 14 MB/s | 52 MB 00:03 2024-05-04T17:42:46.153 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:42:46.157 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:42:46.157 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:42:46.496 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:42:46.496 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:42:48.611 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:42:48.693 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:42:50.472 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:42:50.700 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_ 1/1 2024-05-04T17:42:50.701 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:50.701 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:42:50.701 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:50.701 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:50.701 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:42:50.884 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librbd1-debuginfo 2024-05-04T17:42:51.415 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:08:45 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:42:51.542 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:42:51.542 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:51.542 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:42:51.543 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:51.543 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:42:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 52 M 2024-05-04T17:42:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:51.543 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:42:51.543 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:42:51.543 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:42:51.543 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:42:51.543 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 52 M 2024-05-04T17:42:51.543 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 308 M 2024-05-04T17:42:51.543 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:42:52.416 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:42:52.513 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-2497.gf273489c.el9.x86 1/1 2024-05-04T17:42:54.267 INFO:teuthology.orchestra.run.smithi019.stdout:librbd1-debuginfo-19.0.0-2497.gf273489c.el9.x86 19 MB/s | 52 MB 00:02 2024-05-04T17:42:54.268 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:54.269 INFO:teuthology.orchestra.run.smithi019.stdout:Total 19 MB/s | 52 MB 00:02 2024-05-04T17:42:54.269 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:42:54.273 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:42:54.273 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:42:54.492 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-2497.gf273489c.el9.x86 1/1 2024-05-04T17:42:54.634 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:42:54.634 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:42:54.705 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-2497.gf273489c.el9.x86 1/1 2024-05-04T17:42:54.706 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:54.706 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:42:54.706 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:54.706 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:54.706 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:42:54.883 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install rbd-nbd-debuginfo 2024-05-04T17:42:55.394 INFO:teuthology.orchestra.run.smithi186.stdout:Last metadata expiration check: 0:08:47 ago on Sat 04 May 2024 05:34:08 PM UTC. 2024-05-04T17:42:55.515 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-04T17:42:55.516 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:55.516 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repo Size 2024-05-04T17:42:55.516 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:55.516 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-04T17:42:55.516 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 1.3 M 2024-05-04T17:42:55.516 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:55.516 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-04T17:42:55.517 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-05-04T17:42:55.517 INFO:teuthology.orchestra.run.smithi186.stdout:Install 1 Package 2024-05-04T17:42:55.517 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:55.517 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 1.3 M 2024-05-04T17:42:55.517 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 5.7 M 2024-05-04T17:42:55.517 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-04T17:42:55.822 INFO:teuthology.orchestra.run.smithi186.stdout:rbd-nbd-debuginfo-19.0.0-2497.gf273489c.el9.x86 4.2 MB/s | 1.3 MB 00:00 2024-05-04T17:42:55.823 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:42:55.823 INFO:teuthology.orchestra.run.smithi186.stdout:Total 4.2 MB/s | 1.3 MB 00:00 2024-05-04T17:42:55.823 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-04T17:42:55.826 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-04T17:42:55.827 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-04T17:42:55.871 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-04T17:42:55.871 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-04T17:42:56.110 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-04T17:42:56.202 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:42:56.550 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:42:56.773 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:42:56.773 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:56.773 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-04T17:42:56.773 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:42:56.773 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:42:56.774 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-04T17:42:56.950 DEBUG:teuthology.parallel:result is None 2024-05-04T17:43:01.021 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:43:01.117 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librbd1-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:43:03.213 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:43:03.445 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:43:03.445 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:03.445 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:43:03.446 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:43:03.446 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:03.446 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:43:03.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-cephfs-debuginfo 2024-05-04T17:43:04.176 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:08:58 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:43:04.303 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:43:04.303 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:04.304 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:43:04.304 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:04.304 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:43:04.304 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 615 k 2024-05-04T17:43:04.304 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:04.304 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:43:04.304 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:04.304 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:43:04.304 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:04.304 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 615 k 2024-05-04T17:43:04.304 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 2.1 M 2024-05-04T17:43:04.304 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:43:04.754 INFO:teuthology.orchestra.run.smithi019.stdout:python3-cephfs-debuginfo-19.0.0-2497.gf273489c. 1.3 MB/s | 615 kB 00:00 2024-05-04T17:43:04.754 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:43:04.755 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.3 MB/s | 615 kB 00:00 2024-05-04T17:43:04.755 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:43:04.758 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:43:04.758 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:43:04.801 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:43:04.801 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:43:04.976 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:43:05.079 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-2497.gf273489c.el9 1/1 2024-05-04T17:43:05.444 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-2497.gf273489c.el9 1/1 2024-05-04T17:43:05.670 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-2497.gf273489c.el9 1/1 2024-05-04T17:43:05.670 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:05.670 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:43:05.670 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:43:05.670 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:05.670 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:43:05.864 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rados-debuginfo 2024-05-04T17:43:06.399 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:09:00 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:43:06.526 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:43:06.526 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:06.527 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:43:06.527 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:06.527 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:43:06.527 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 770 k 2024-05-04T17:43:06.527 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:06.527 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:43:06.527 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:06.527 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:43:06.527 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:06.527 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 770 k 2024-05-04T17:43:06.527 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 2.7 M 2024-05-04T17:43:06.527 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:43:06.807 INFO:teuthology.orchestra.run.smithi019.stdout:python3-rados-debuginfo-19.0.0-2497.gf273489c.e 2.7 MB/s | 770 kB 00:00 2024-05-04T17:43:06.808 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:43:06.808 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.7 MB/s | 770 kB 00:00 2024-05-04T17:43:06.808 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:43:06.812 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:43:06.812 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:43:06.855 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:43:06.856 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:43:07.060 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:43:07.157 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-debuginfo-2:19.0.0-2497.gf273489c.el9. 1/1 2024-05-04T17:43:07.477 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-2497.gf273489c.el9. 1/1 2024-05-04T17:43:07.699 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-2497.gf273489c.el9. 1/1 2024-05-04T17:43:07.700 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:07.700 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:43:07.700 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:43:07.700 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:07.700 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:43:07.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rbd-debuginfo 2024-05-04T17:43:08.408 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:09:02 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:43:08.533 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:43:08.534 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:08.534 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:43:08.534 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:08.534 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:43:08.534 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 1.1 M 2024-05-04T17:43:08.534 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:08.534 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:43:08.534 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:08.535 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:43:08.535 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:08.535 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 1.1 M 2024-05-04T17:43:08.535 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 4.3 M 2024-05-04T17:43:08.535 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:43:08.976 INFO:teuthology.orchestra.run.smithi019.stdout:python3-rbd-debuginfo-19.0.0-2497.gf273489c.el9 2.5 MB/s | 1.1 MB 00:00 2024-05-04T17:43:08.977 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:43:08.977 INFO:teuthology.orchestra.run.smithi019.stdout:Total 2.5 MB/s | 1.1 MB 00:00 2024-05-04T17:43:08.977 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:43:08.981 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:43:08.981 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:43:09.026 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:43:09.027 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:43:09.266 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:43:09.368 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:43:09.696 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:43:09.927 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:43:09.927 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:09.927 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:43:09.927 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:43:09.927 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:09.927 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:43:10.116 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rgw-debuginfo 2024-05-04T17:43:10.647 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:09:04 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:43:10.772 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:43:10.772 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:10.772 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:43:10.772 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:10.772 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:43:10.772 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 337 k 2024-05-04T17:43:10.773 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:10.773 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:43:10.773 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:10.773 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:43:10.773 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:10.773 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 337 k 2024-05-04T17:43:10.773 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.0 M 2024-05-04T17:43:10.774 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:43:11.021 INFO:teuthology.orchestra.run.smithi019.stdout:python3-rgw-debuginfo-19.0.0-2497.gf273489c.el9 1.3 MB/s | 337 kB 00:00 2024-05-04T17:43:11.022 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:43:11.022 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.3 MB/s | 337 kB 00:00 2024-05-04T17:43:11.022 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:43:11.026 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:43:11.026 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:43:11.065 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:43:11.065 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:43:11.204 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:43:11.303 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:43:11.642 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:43:11.878 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x8 1/1 2024-05-04T17:43:11.879 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:11.879 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:43:11.879 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:43:11.879 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:11.879 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:43:12.069 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-fuse-debuginfo 2024-05-04T17:43:12.602 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:09:06 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:43:12.728 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 516 k 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 516 k 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 2.6 M 2024-05-04T17:43:12.729 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:43:13.008 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-fuse-debuginfo-19.0.0-2497.gf273489c.el9.x8 1.8 MB/s | 516 kB 00:00 2024-05-04T17:43:13.008 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:43:13.008 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.8 MB/s | 516 kB 00:00 2024-05-04T17:43:13.008 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:43:13.012 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:43:13.012 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:43:13.052 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:43:13.053 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:43:13.222 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:43:13.324 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:43:13.674 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:43:13.893 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_6 1/1 2024-05-04T17:43:13.893 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:13.893 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:43:13.894 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:43:13.894 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:13.894 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:43:14.063 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-mirror-debuginfo 2024-05-04T17:43:14.601 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:09:08 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:43:14.726 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:43:14.727 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:14.727 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:43:14.727 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:14.727 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:43:14.727 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 52 M 2024-05-04T17:43:14.727 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:14.727 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:43:14.727 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:14.727 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:43:14.728 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:14.728 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 52 M 2024-05-04T17:43:14.728 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 298 M 2024-05-04T17:43:14.728 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:43:17.402 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-mirror-debuginfo-19.0.0-2497.gf273489c.el9. 19 MB/s | 52 MB 00:02 2024-05-04T17:43:17.403 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:43:17.403 INFO:teuthology.orchestra.run.smithi019.stdout:Total 19 MB/s | 52 MB 00:02 2024-05-04T17:43:17.404 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:43:17.408 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:43:17.408 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:43:17.764 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:43:17.764 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:43:24.033 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:43:24.134 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-2497.gf273489c.el9.x86 1/1 2024-05-04T17:43:26.278 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-2497.gf273489c.el9.x86 1/1 2024-05-04T17:43:26.497 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-2497.gf273489c.el9.x86 1/1 2024-05-04T17:43:26.498 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:26.498 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:43:26.498 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:43:26.498 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:26.498 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:43:26.674 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-nbd-debuginfo 2024-05-04T17:43:27.214 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:09:21 ago on Sat 04 May 2024 05:34:06 PM UTC. 2024-05-04T17:43:27.341 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-05-04T17:43:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:27.342 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-05-04T17:43:27.342 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-05-04T17:43:27.343 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-2497.gf273489c.el9 ceph 1.3 M 2024-05-04T17:43:27.343 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-05-04T17:43:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-05-04T17:43:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-05-04T17:43:27.343 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 1.3 M 2024-05-04T17:43:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 5.7 M 2024-05-04T17:43:27.343 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-05-04T17:43:28.029 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-nbd-debuginfo-19.0.0-2497.gf273489c.el9.x86 1.9 MB/s | 1.3 MB 00:00 2024-05-04T17:43:28.030 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-05-04T17:43:28.030 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.9 MB/s | 1.3 MB 00:00 2024-05-04T17:43:28.030 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-05-04T17:43:28.034 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-05-04T17:43:28.034 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-05-04T17:43:28.081 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-05-04T17:43:28.082 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-05-04T17:43:28.335 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-05-04T17:43:28.445 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:43:28.792 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:43:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 1/1 2024-05-04T17:43:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:29.025 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-05-04T17:43:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-debuginfo-2:19.0.0-2497.gf273489c.el9.x86_64 2024-05-04T17:43:29.025 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:29.025 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-05-04T17:43:29.230 DEBUG:teuthology.parallel:result is None 2024-05-04T17:43:29.230 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:43:29.373 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-04T17:43:29.394 INFO:teuthology.orchestra.run.smithi019.stdout:19.0.0-2497.gf273489c.el9 2024-05-04T17:43:29.395 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2497.gf273489c.el9 2024-05-04T17:43:29.395 INFO:teuthology.task.install:The correct ceph version 19.0.0-2497.gf273489c is installed. 2024-05-04T17:43:29.396 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:43:29.533 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-04T17:43:29.557 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-2497.gf273489c.el9 2024-05-04T17:43:29.557 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2497.gf273489c.el9 2024-05-04T17:43:29.557 INFO:teuthology.task.install:The correct ceph version 19.0.0-2497.gf273489c is installed. 2024-05-04T17:43:29.559 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-04T17:43:29.560 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:43:29.560 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-04T17:43:29.585 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:43:29.585 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-04T17:43:29.624 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-04T17:43:29.625 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:43:29.625 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/daemon-helper 2024-05-04T17:43:29.650 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-04T17:43:29.714 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:43:29.714 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-05-04T17:43:29.743 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-04T17:43:29.811 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-04T17:43:29.811 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:43:29.812 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-04T17:43:29.837 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-04T17:43:29.902 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:43:29.902 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-04T17:43:29.928 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-04T17:43:29.994 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-04T17:43:29.995 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:43:29.995 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/stdin-killer 2024-05-04T17:43:30.019 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-04T17:43:30.084 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:43:30.084 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-05-04T17:43:30.109 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-04T17:43:30.173 INFO:teuthology.run_tasks:Running task ceph... 2024-05-04T17:43:30.258 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-04T17:43:30.258 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /var/log/ceph 2024-05-04T17:43:30.260 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /var/log/ceph 2024-05-04T17:43:30.284 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-04T17:43:30.285 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-04T17:43:30.325 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-04T17:43:30.349 INFO:tasks.ceph:Creating extra log directories... 2024-05-04T17:43:30.349 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-04T17:43:30.389 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-04T17:43:30.422 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-04T17:43:30.422 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-04T17:43:30.423 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124', 'branch': 'wip-yuriw11-testing-20240501.200505-squid', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7687124', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuriw11-testing-20240501.200505-squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuriw11-testing-20240501.200505-squid', 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8797, 'sha1': 'f273489c6dcd4bc88409993babd09dd99491162c', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuriw11-testing-20240501.200505-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f273489c6dcd4bc88409993babd09dd99491162c', 'targets': {'smithi019.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOvty0GMgfRqiueBK+9zrw9w+jUpACO7qhUu4rXcOfp2gxDXptEzxgA9fTtDwyweBBm7WSF980wwDx3iJ6RO9Bw=', 'smithi186.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHYzvai/rL2inX2fbEzwz6S4DYA4Fbh2e/rKxOZaVRoBX4sR4J8ehcnFxUBq0G36PrKUFqP287tjivJTsIZ7S2w='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-02_23:59:28', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-04T17:43:30.423 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-04T17:43:30.453 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-04T17:43:30.468 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/run/ceph 2024-05-04T17:43:30.508 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/run/ceph 2024-05-04T17:43:30.531 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:43:30.531 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2024-05-04T17:43:30.585 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-04T17:43:30.585 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2024-05-04T17:43:30.641 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-04T17:43:30.641 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:43:30.641 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-04T17:43:30.641 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:43:30.641 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:43:30.641 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-05-04 17:36:04.938298235 +0000 2024-05-04T17:43:30.641 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-05-04 17:33:28.103668312 +0000 2024-05-04T17:43:30.641 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-05-04 17:33:28.103668312 +0000 2024-05-04T17:43:30.641 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-05-04 17:33:28.103668312 +0000 2024-05-04T17:43:30.641 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-04T17:43:30.703 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-05-04T17:43:30.703 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-05-04T17:43:30.703 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000241557 s, 2.1 MB/s 2024-05-04T17:43:30.705 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-04T17:43:30.760 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2024-05-04T17:43:30.816 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-04T17:43:30.816 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:43:30.816 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-04T17:43:30.816 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:43:30.816 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:43:30.816 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-05-04 17:36:04.939298224 +0000 2024-05-04T17:43:30.816 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-05-04 17:33:28.433664815 +0000 2024-05-04T17:43:30.816 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-05-04 17:33:28.433664815 +0000 2024-05-04T17:43:30.816 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-05-04 17:33:28.433664815 +0000 2024-05-04T17:43:30.817 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-04T17:43:30.878 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-05-04T17:43:30.878 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-05-04T17:43:30.878 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000287246 s, 1.8 MB/s 2024-05-04T17:43:30.880 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-04T17:43:30.935 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2024-05-04T17:43:30.990 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-04T17:43:30.990 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:43:30.991 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-04T17:43:30.991 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:43:30.991 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:43:30.991 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-05-04 17:36:04.939298224 +0000 2024-05-04T17:43:30.991 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-05-04 17:33:28.785661085 +0000 2024-05-04T17:43:30.991 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-05-04 17:33:28.785661085 +0000 2024-05-04T17:43:30.991 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-05-04 17:33:28.785661085 +0000 2024-05-04T17:43:30.991 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-04T17:43:31.053 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-05-04T17:43:31.053 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-05-04T17:43:31.053 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000189237 s, 2.7 MB/s 2024-05-04T17:43:31.054 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-04T17:43:31.110 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2024-05-04T17:43:31.166 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-04T17:43:31.166 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:43:31.166 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-05-04T17:43:31.166 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:43:31.166 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:43:31.166 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-05-04 17:36:04.939298224 +0000 2024-05-04T17:43:31.166 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-05-04 17:33:29.124657493 +0000 2024-05-04T17:43:31.166 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-05-04 17:33:29.124657493 +0000 2024-05-04T17:43:31.166 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-05-04 17:33:29.124657493 +0000 2024-05-04T17:43:31.167 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-04T17:43:31.228 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-05-04T17:43:31.228 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-05-04T17:43:31.229 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000169159 s, 3.0 MB/s 2024-05-04T17:43:31.230 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-04T17:43:31.285 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-04T17:43:31.285 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:43:31.285 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-05-04T17:43:31.302 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-04T17:43:31.302 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-05-04T17:43:31.359 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-04T17:43:31.359 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:43:31.359 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-05-04T17:43:31.359 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:43:31.359 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:43:31.359 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-04 17:35:59.796249292 +0000 2024-05-04T17:43:31.359 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-04 17:33:23.568501196 +0000 2024-05-04T17:43:31.359 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-04 17:33:23.568501196 +0000 2024-05-04T17:43:31.359 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-04 17:33:23.568501196 +0000 2024-05-04T17:43:31.360 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-04T17:43:31.422 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-04T17:43:31.422 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-04T17:43:31.422 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000215349 s, 2.4 MB/s 2024-05-04T17:43:31.423 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-04T17:43:31.480 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-05-04T17:43:31.536 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-04T17:43:31.536 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:43:31.537 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-04T17:43:31.537 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:43:31.537 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:43:31.537 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-04 17:35:59.796249292 +0000 2024-05-04T17:43:31.537 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-04 17:33:23.922494895 +0000 2024-05-04T17:43:31.537 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-04 17:33:23.922494895 +0000 2024-05-04T17:43:31.537 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-04 17:33:23.922494895 +0000 2024-05-04T17:43:31.537 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-04T17:43:31.599 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-04T17:43:31.599 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-04T17:43:31.600 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.00019219 s, 2.7 MB/s 2024-05-04T17:43:31.601 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-04T17:43:31.657 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-05-04T17:43:31.713 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-04T17:43:31.713 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:43:31.713 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-05-04T17:43:31.713 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:43:31.713 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:43:31.713 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-04 17:35:59.796249292 +0000 2024-05-04T17:43:31.713 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-04 17:33:24.271488682 +0000 2024-05-04T17:43:31.713 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-04 17:33:24.271488682 +0000 2024-05-04T17:43:31.714 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-04 17:33:24.271488682 +0000 2024-05-04T17:43:31.714 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-04T17:43:31.776 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-04T17:43:31.776 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-04T17:43:31.776 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.00020966 s, 2.4 MB/s 2024-05-04T17:43:31.777 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-04T17:43:31.833 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-05-04T17:43:31.889 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-04T17:43:31.889 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-04T17:43:31.889 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-04T17:43:31.889 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-04T17:43:31.889 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-04T17:43:31.889 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-04 17:35:59.797249274 +0000 2024-05-04T17:43:31.889 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-04 17:33:24.619482487 +0000 2024-05-04T17:43:31.889 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-04 17:33:24.619482487 +0000 2024-05-04T17:43:31.889 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-04 17:33:24.619482487 +0000 2024-05-04T17:43:31.890 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-04T17:43:31.950 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-04T17:43:31.951 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-04T17:43:31.951 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000163607 s, 3.1 MB/s 2024-05-04T17:43:31.952 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-04T17:43:32.007 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-04T17:43:32.007 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-04T17:43:32.007 INFO:tasks.ceph:Generating config... 2024-05-04T17:43:32.009 INFO:tasks.ceph:[client] debug monc = 20 2024-05-04T17:43:32.009 INFO:tasks.ceph:[client] debug ms = 1 2024-05-04T17:43:32.009 INFO:tasks.ceph:[client] debug objecter = 20 2024-05-04T17:43:32.009 INFO:tasks.ceph:[client] debug rados = 20 2024-05-04T17:43:32.009 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-05-04T17:43:32.009 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-05-04T17:43:32.009 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-05-04T17:43:32.009 INFO:tasks.ceph:[global] ms type = async 2024-05-04T17:43:32.009 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-05-04T17:43:32.009 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-04T17:43:32.009 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-04T17:43:32.009 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-04T17:43:32.010 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-04T17:43:32.010 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-04T17:43:32.010 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-05-04T17:43:32.010 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-05-04T17:43:32.010 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-05-04T17:43:32.010 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-05-04T17:43:32.010 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-05-04T17:43:32.010 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-05-04T17:43:32.010 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-05-04T17:43:32.010 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-05-04T17:43:32.010 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-04T17:43:32.010 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-04T17:43:32.010 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-04T17:43:32.011 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-04T17:43:32.011 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-05-04T17:43:32.011 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-05-04T17:43:32.011 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-05-04T17:43:32.011 INFO:tasks.ceph:[osd] debug monc = 20 2024-05-04T17:43:32.011 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-04T17:43:32.011 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-04T17:43:32.011 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-05-04T17:43:32.011 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-04T17:43:32.011 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-04T17:43:32.011 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-04T17:43:32.011 INFO:tasks.ceph:[osd] osd class default list = * 2024-05-04T17:43:32.012 INFO:tasks.ceph:[osd] osd class load list = * 2024-05-04T17:43:32.012 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-05-04T17:43:32.012 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-05-04T17:43:32.012 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-05-04T17:43:32.012 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-04T17:43:32.012 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-05-04T17:43:32.012 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-05-04T17:43:32.012 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-05-04T17:43:32.012 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-05-04T17:43:32.012 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-04T17:43:32.012 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-05-04T17:43:32.012 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-05-04T17:43:32.013 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-05-04T17:43:32.013 INFO:tasks.ceph:Setting up mon.a... 2024-05-04T17:43:32.013 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-04T17:43:32.062 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.keyring 2024-05-04T17:43:32.066 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-04T17:43:32.110 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-04T17:43:32.175 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.19'), ('mon.c', '[v2:172.21.15.19:3301,v1:172.21.15.19:6790]'), ('mon.b', '172.21.15.186')] 2024-05-04T17:43:32.176 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.19,[v2:172.21.15.19:3301,v1:172.21.15.19:6790],172.21.15.186', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-05-04T17:43:32.177 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:43:32.177 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-04T17:43:32.231 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.19 --addv c '[v2:172.21.15.19:3301,v1:172.21.15.19:6790]' --add b 172.21.15.186 --print /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:43:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:43:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: generated fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:43:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:setting min_mon_release = quincy 2024-05-04T17:43:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:epoch 0 2024-05-04T17:43:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:43:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:43:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:43:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:min_mon_release 17 (quincy) 2024-05-04T17:43:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:election_strategy: 1 2024-05-04T17:43:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:43:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:43:32.318 INFO:teuthology.orchestra.run.smithi019.stdout:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:43:32.319 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-04T17:43:32.320 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-04T17:43:32.334 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 18ced2df-a29f-46cc-b71a-1855bfb60c47... 2024-05-04T17:43:32.335 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-04T17:43:32.377 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-04T17:43:32.415 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2024-05-04T17:43:32.415 INFO:teuthology.orchestra.run.smithi019.stdout: chdir = "" 2024-05-04T17:43:32.415 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: auth supported = cephx 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: filestore xattr use omap = true 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: mon clock drift allowed = 1.000 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: osd crush chooseleaf type = 0 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: auth debug = true 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on old message = true 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on bug = true 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: mon pg warn max object skew = 0 2024-05-04T17:43:32.416 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: # disable pg_autoscaler by default for new pools 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default size = 2 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow primary affinity = true 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow pg remap = true 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on legacy crush tunables = false 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on crush straw calc version zero = false 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on no sortbitwise = false 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on osd down out interval zero = false 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on too few osds = false 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: mon_allow_pool_size_one = true 2024-05-04T17:43:32.417 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: osd default data pool replay window = 5 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: mon allow pool delete = true 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: mon cluster log file level = debug 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: debug asserts on shutdown = true 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: mon health detail to clog = false 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: mon host = "172.21.15.19,[v2:172.21.15.19:3301,v1:172.21.15.19:6790],172.21.15.186" 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: mon client directed command retry = 5 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: mon election default strategy = 1 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: ms inject socket failures = 5000 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: ms type = async 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: osd heartbeat grace = 80 2024-05-04T17:43:32.418 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout:[osd] 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: osd journal size = 100 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub load threshold = 5.0 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub max interval = 600 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock profile = high_recovery_ops 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: osd recover clone overlap = true 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: osd recovery max chunk = 1048576 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug shutdown = true 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug op order = true 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify stray on activate = true 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug trim objects = true 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.419 INFO:teuthology.orchestra.run.smithi019.stdout: osd open classes on start = true 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug pg log writeout = true 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: osd deep scrub update digest min age = 30 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: osd map max advance = 10 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: journal zero on create = true 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: filestore ondisk finisher threads = 3 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: filestore apply finisher threads = 3 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: bdev debug aio = true 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug misdirected ops = true 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: bdev async discard = True 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: bdev enable discard = True 2024-05-04T17:43:32.420 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore block size = 96636764160 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore fsck on mount = True 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore zero block detection = True 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluefs = 1 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluestore = 1 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: debug monc = 20 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 20 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: debug rocksdb = 10 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd backfillfull_ratio = 0.85 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd full ratio = 0.9 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd nearfull ratio = 0.8 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: osd class default list = * 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: osd class load list = * 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: osd client watch timeout = 120 2024-05-04T17:43:32.421 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify cached snaps = True 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify missing on start = True 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: osd failsafe full ratio = 0.95 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: osd fast shutdown = False 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock override recovery settings = True 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: osd memory target = 1610612736 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: osd objectstore = bluestore 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: osd op queue = debug_random 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: osd op queue cut off = debug_random 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: osd shutdown pgref assert = True 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr] 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 20 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-05-04T17:43:32.422 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: mgr/telemetry/nag = false 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout:[mon] 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 20 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: mon data avail warn = 5 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: mon mgr mkfs grace = 240 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd reporter subtree level = osd 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd prime pg temp = true 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-05-04T17:43:32.423 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: auth mon ticket ttl = 660 # 11m 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: auth service ticket ttl = 240 # 4m 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: # don't complain about insecure global_id in the test suite 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: # 1m isn't quite enough 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: mon_down_mkfs_grace = 2m 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_filestore_osds = false 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: mon min osdmap epochs = 50 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd crush smoke test = False 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: mon osdmap full prune interval = 2 2024-05-04T17:43:32.424 INFO:teuthology.orchestra.run.smithi019.stdout: mon osdmap full prune min = 15 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: mon osdmap full prune txsize = 2 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: mon scrub interval = 300 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on pool no app = False 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: paxos service trim min = 10 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout:[client] 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: rgw cache enabled = true 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable ops log = true 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable usage log = true 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: debug monc = 20 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: debug objecter = 20 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout: debug rados = 20 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.a] 2024-05-04T17:43:32.425 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.c] 2024-05-04T17:43:32.426 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.b] 2024-05-04T17:43:32.426 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2024-05-04T17:43:32.426 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2024-05-04T17:43:32.426 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-04T17:43:32.426 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = cephx 2024-05-04T17:43:32.426 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.426 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2024-05-04T17:43:32.426 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.427 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2024-05-04T17:43:32.428 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = "172.21.15.19,[v2:172.21.15.19:3301,v1:172.21.15.19:6790],172.21.15.186" 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon client directed command retry = 5 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: mon election default strategy = 1 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject socket failures = 5000 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: ms type = async 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: osd heartbeat grace = 80 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2024-05-04T17:43:32.429 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 600 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock profile = high_recovery_ops 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug trim objects = true 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2024-05-04T17:43:32.430 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: bdev async discard = True 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: bdev enable discard = True 2024-05-04T17:43:32.431 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore zero block detection = True 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 1 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 1 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug monc = 20 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 20 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 10 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: osd class default list = * 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: osd class load list = * 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: osd client watch timeout = 120 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify cached snaps = True 2024-05-04T17:43:32.432 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify missing on start = True 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd fast shutdown = False 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock override recovery settings = True 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd memory target = 1610612736 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd op queue = debug_random 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd op queue cut off = debug_random 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: osd shutdown pgref assert = True 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-05-04T17:43:32.433 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 20 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.434 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon min osdmap epochs = 50 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd crush smoke test = False 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon osdmap full prune interval = 2 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon osdmap full prune min = 15 2024-05-04T17:43:32.435 INFO:teuthology.orchestra.run.smithi186.stdout: mon osdmap full prune txsize = 2 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: mon scrub interval = 300 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on pool no app = False 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: paxos service trim min = 10 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: debug monc = 20 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: debug objecter = 20 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout: debug rados = 20 2024-05-04T17:43:32.436 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2024-05-04T17:43:32.437 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.c] 2024-05-04T17:43:32.437 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.b] 2024-05-04T17:43:32.438 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-04T17:43:32.438 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-04T17:43:32.532 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-04T17:43:32.532 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:43:32.532 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-04T17:43:32.547 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:43:32.547 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-04T17:43:32.603 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi019.front.sepia.ceph.com 2024-05-04T17:43:32.603 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:43:32.603 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-04T17:43:32.603 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-04T17:43:32.676 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:43:32.676 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:43:32.732 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi186.front.sepia.ceph.com 2024-05-04T17:43:32.732 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:43:32.732 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-04T17:43:32.732 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-04T17:43:32.766 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:43:32.766 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:43:32.824 INFO:tasks.ceph:Setting up mon nodes... 2024-05-04T17:43:32.824 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-04T17:43:32.824 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-04T17:43:32.873 INFO:teuthology.orchestra.run.smithi019.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-04T17:43:32.876 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-04T17:43:32.933 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-04T17:43:32.935 INFO:tasks.ceph:Setting up mds nodes... 2024-05-04T17:43:32.935 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-04T17:43:32.936 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-04T17:43:32.976 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-04T17:43:32.988 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-04T17:43:33.026 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-05-04T17:43:33.038 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-04T17:43:33.039 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-04T17:43:33.039 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-04T17:43:33.063 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-04T17:43:33.063 INFO:tasks.ceph:role: osd.0 2024-05-04T17:43:33.063 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com 2024-05-04T17:43:33.063 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-04T17:43:33.135 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:43:33.135 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:43:33.135 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:43:33.135 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:43:33.135 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:43:33.135 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:43:33.135 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:43:33.136 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:43:33.136 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:43:33.136 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:43:33.637 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-05-04T17:43:33.639 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-05-04T17:43:33.639 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-04T17:43:33.786 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-04T17:43:33.819 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-04T17:43:33.885 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-04T17:43:33.885 INFO:tasks.ceph:role: osd.1 2024-05-04T17:43:33.885 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com 2024-05-04T17:43:33.885 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-04T17:43:33.958 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:43:33.959 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:43:33.959 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:43:33.959 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:43:33.959 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:43:33.959 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:43:33.959 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:43:33.959 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:43:33.959 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:43:33.959 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:43:34.415 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-05-04T17:43:34.417 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-05-04T17:43:34.418 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-04T17:43:34.524 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-04T17:43:34.555 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-04T17:43:34.622 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-04T17:43:34.623 INFO:tasks.ceph:role: osd.2 2024-05-04T17:43:34.623 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com 2024-05-04T17:43:34.623 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-04T17:43:34.696 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:43:34.696 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:43:34.696 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:43:34.696 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:43:34.696 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:43:34.696 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:43:34.696 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:43:34.697 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:43:34.697 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:43:34.697 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:43:35.192 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-05-04T17:43:35.194 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-05-04T17:43:35.194 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-04T17:43:35.347 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-04T17:43:35.380 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-04T17:43:35.448 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-04T17:43:35.448 INFO:tasks.ceph:role: osd.3 2024-05-04T17:43:35.448 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi019.front.sepia.ceph.com 2024-05-04T17:43:35.448 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-04T17:43:35.522 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:43:35.522 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:43:35.522 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:43:35.522 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:43:35.522 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:43:35.522 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:43:35.522 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:43:35.522 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:43:35.522 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:43:35.522 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:43:36.027 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-05-04T17:43:36.029 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-05-04T17:43:36.029 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-04T17:43:36.174 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-04T17:43:36.212 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:43:36.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:36.308+0000 7f4961223640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-04T17:43:36.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:36.309+0000 7f4961223640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-04T17:43:36.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:36.309+0000 7f4961223640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-04T17:43:36.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:36.309+0000 7f4961223640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-04T17:43:36.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:36.309+0000 7f4961223640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-04T17:43:36.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:36.309+0000 7f4961223640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-04T17:43:39.160 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-04T17:43:39.189 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:43:39.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:39.278+0000 7efc55643640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-04T17:43:39.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:39.279+0000 7efc55643640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-04T17:43:39.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:39.279+0000 7efc55643640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-04T17:43:39.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:39.279+0000 7efc55643640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-04T17:43:39.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:39.279+0000 7efc55643640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-04T17:43:39.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:39.279+0000 7efc55643640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-04T17:43:42.121 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-04T17:43:42.150 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:43:42.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:42.242+0000 7fc3c0506640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-04T17:43:42.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:42.242+0000 7fc3c0506640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-04T17:43:42.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:42.242+0000 7fc3c0506640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-04T17:43:42.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:42.242+0000 7fc3c0506640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-04T17:43:42.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:42.242+0000 7fc3c0506640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-04T17:43:42.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:42.242+0000 7fc3c0506640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-04T17:43:45.092 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-04T17:43:45.121 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:43:45.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:45.209+0000 7f1b5f015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-04T17:43:45.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:45.209+0000 7f1b5f015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-04T17:43:45.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:45.209+0000 7f1b5f015640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-04T17:43:45.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:45.209+0000 7f1b5f015640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-04T17:43:45.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:45.209+0000 7f1b5f015640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-04T17:43:45.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:43:45.209+0000 7f1b5f015640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-04T17:43:48.051 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-04T17:43:48.080 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-04T17:43:48.108 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-04T17:43:48.109 INFO:tasks.ceph:role: osd.4 2024-05-04T17:43:48.109 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-04T17:43:48.109 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-04T17:43:48.184 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:43:48.184 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:43:48.184 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:43:48.184 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:43:48.184 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:43:48.184 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:43:48.184 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:43:48.185 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:43:48.185 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:43:48.185 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:43:48.696 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-05-04T17:43:48.698 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-05-04T17:43:48.698 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-04T17:43:48.819 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-04T17:43:48.853 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-04T17:43:48.920 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-04T17:43:48.920 INFO:tasks.ceph:role: osd.5 2024-05-04T17:43:48.920 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-04T17:43:48.920 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-04T17:43:48.992 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:43:48.992 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:43:48.992 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:43:48.993 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:43:48.993 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:43:48.993 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:43:48.993 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:43:48.993 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:43:48.993 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:43:48.993 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:43:49.451 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-05-04T17:43:49.454 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-05-04T17:43:49.454 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-04T17:43:49.549 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-04T17:43:49.580 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-04T17:43:49.645 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-04T17:43:49.645 INFO:tasks.ceph:role: osd.6 2024-05-04T17:43:49.646 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-04T17:43:49.646 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-04T17:43:49.718 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:43:49.719 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:43:49.719 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:43:49.719 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:43:49.719 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:43:49.719 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:43:49.719 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:43:49.719 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:43:49.719 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:43:49.719 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:43:50.223 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-05-04T17:43:50.225 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-05-04T17:43:50.225 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-04T17:43:50.378 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-04T17:43:50.411 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-04T17:43:50.478 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-04T17:43:50.478 INFO:tasks.ceph:role: osd.7 2024-05-04T17:43:50.478 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-04T17:43:50.478 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-04T17:43:50.552 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-04T17:43:50.552 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-04T17:43:50.552 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-04T17:43:50.553 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-04T17:43:50.553 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-04T17:43:50.553 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-05-04T17:43:50.553 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-04T17:43:50.553 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-04T17:43:50.553 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-04T17:43:50.553 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-04T17:43:51.065 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-05-04T17:43:51.067 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-05-04T17:43:51.067 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-04T17:43:51.190 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-04T17:43:51.221 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:43:51.313 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:51.312+0000 7f39fbf99640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-04T17:43:51.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:51.312+0000 7f39fbf99640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-04T17:43:51.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:51.313+0000 7f39fbf99640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-04T17:43:51.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:51.313+0000 7f39fbf99640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-04T17:43:51.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:51.313+0000 7f39fbf99640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-04T17:43:51.314 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:51.313+0000 7f39fbf99640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-04T17:43:54.159 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-04T17:43:54.187 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:43:54.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:54.274+0000 7f3e29a43640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-04T17:43:54.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:54.274+0000 7f3e29a43640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-04T17:43:54.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:54.275+0000 7f3e29a43640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-04T17:43:54.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:54.275+0000 7f3e29a43640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-04T17:43:54.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:54.275+0000 7f3e29a43640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-04T17:43:54.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:54.275+0000 7f3e29a43640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-04T17:43:57.115 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-04T17:43:57.143 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:43:57.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:57.229+0000 7f25c3bd8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-04T17:43:57.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:57.229+0000 7f25c3bd8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-04T17:43:57.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:57.229+0000 7f25c3bd8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-04T17:43:57.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:57.229+0000 7f25c3bd8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-04T17:43:57.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:57.229+0000 7f25c3bd8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-04T17:43:57.231 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:43:57.230+0000 7f25c3bd8640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-04T17:44:00.071 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-04T17:44:00.099 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:44:00.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:00.185+0000 7f5f4a830640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-04T17:44:00.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:00.185+0000 7f5f4a830640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-04T17:44:00.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:00.185+0000 7f5f4a830640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-04T17:44:00.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:00.185+0000 7f5f4a830640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-04T17:44:00.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:00.185+0000 7f5f4a830640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-04T17:44:00.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:00.186+0000 7f5f4a830640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-04T17:44:03.036 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-04T17:44:03.064 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-04T17:44:03.064 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:44:03.064 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-04T17:44:03.095 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:44:03.095 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-04T17:44:03.162 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:44:03.163 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-04T17:44:03.231 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:44:03.232 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-04T17:44:03.299 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:44:03.299 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-04T17:44:03.366 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:44:03.367 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-04T17:44:03.395 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:44:03.395 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-04T17:44:03.462 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:44:03.462 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-04T17:44:03.529 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:44:03.529 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-04T17:44:03.595 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:44:03.595 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-04T17:44:03.660 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:44:03.660 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-04T17:44:03.677 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:44:03.677 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-05-04T17:44:03.716 INFO:tasks.ceph:Adding keys to all mons... 2024-05-04T17:44:03.717 DEBUG:teuthology.orchestra.run.smithi019:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-04T17:44:03.720 DEBUG:teuthology.orchestra.run.smithi186:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-04T17:44:03.759 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.y] 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDEczZmlzIZNBAAEtKhLyLtOm0jDQgBomwASw== 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.0] 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDIczZmHyp3EhAAHHsFHYMJPj41yOAAVruJtg== 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.1] 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDLczZmT5mpEBAAIvF5hY5ZMJ2Cw3cUUr9xsA== 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.2] 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDOczZmnMh5DhAAACN/lkrH4e5V01tIz2ShNw== 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.3] 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDRczZmkL6DDBAA+vKzNC9jhP/u6YwUixhxaA== 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.x] 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDEczZmhSChNxAAiroU/hioejo7uqzqb8tLpg== 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.4] 2024-05-04T17:44:03.760 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDXczZmCoywEhAAZkZ8StHeVu9/B1TCSGdXmw== 2024-05-04T17:44:03.761 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.5] 2024-05-04T17:44:03.761 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDaczZmY0VtEBAA/achzEIW52guWCd1SK+Jag== 2024-05-04T17:44:03.761 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.6] 2024-05-04T17:44:03.761 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDdczZmxDK6DRAAg/WyyHiXdxwBRv8z172TLg== 2024-05-04T17:44:03.761 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.7] 2024-05-04T17:44:03.761 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDgczZmbhouCxAAAwumGjyxbAMAoOCmEzhXLQ== 2024-05-04T17:44:03.761 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2024-05-04T17:44:03.761 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDEczZmzFgwOhAAvfikESZ4PuFktwjOohxvLA== 2024-05-04T17:44:03.761 INFO:teuthology.orchestra.run.smithi019.stdout:[client.1] 2024-05-04T17:44:03.761 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQDFczZmA4KYARAA8rApfEPOaAsgmJMK9iPzGQ== 2024-05-04T17:44:03.779 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.y] 2024-05-04T17:44:03.779 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDEczZmlzIZNBAAEtKhLyLtOm0jDQgBomwASw== 2024-05-04T17:44:03.779 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.0] 2024-05-04T17:44:03.779 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDIczZmHyp3EhAAHHsFHYMJPj41yOAAVruJtg== 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.1] 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDLczZmT5mpEBAAIvF5hY5ZMJ2Cw3cUUr9xsA== 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.2] 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDOczZmnMh5DhAAACN/lkrH4e5V01tIz2ShNw== 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.3] 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDRczZmkL6DDBAA+vKzNC9jhP/u6YwUixhxaA== 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.x] 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDEczZmhSChNxAAiroU/hioejo7uqzqb8tLpg== 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.4] 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDXczZmCoywEhAAZkZ8StHeVu9/B1TCSGdXmw== 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.5] 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDaczZmY0VtEBAA/achzEIW52guWCd1SK+Jag== 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.6] 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDdczZmxDK6DRAAg/WyyHiXdxwBRv8z172TLg== 2024-05-04T17:44:03.780 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.7] 2024-05-04T17:44:03.781 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDgczZmbhouCxAAAwumGjyxbAMAoOCmEzhXLQ== 2024-05-04T17:44:03.781 INFO:teuthology.orchestra.run.smithi186.stdout:[client.0] 2024-05-04T17:44:03.781 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDEczZmzFgwOhAAvfikESZ4PuFktwjOohxvLA== 2024-05-04T17:44:03.781 INFO:teuthology.orchestra.run.smithi186.stdout:[client.1] 2024-05-04T17:44:03.781 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDFczZmA4KYARAA8rApfEPOaAsgmJMK9iPzGQ== 2024-05-04T17:44:03.782 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-04T17:44:03.806 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-04T17:44:03.872 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:03.875 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:03.924 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:03.926 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:03.976 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:03.978 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:04.026 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:04.028 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:04.077 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-04T17:44:04.079 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-04T17:44:04.128 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:04.130 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:04.179 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:04.181 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:04.229 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:04.231 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:04.280 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:04.282 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-04T17:44:04.330 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-04T17:44:04.332 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-04T17:44:04.384 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-04T17:44:04.386 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-04T17:44:04.438 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-04T17:44:04.439 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-04T17:44:04.463 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-04T17:44:04.557 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-04T17:44:04.581 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-04T17:44:04.646 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-04T17:44:04.740 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-04T17:44:04.764 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-04T17:44:04.788 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-04T17:44:04.881 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-04T17:44:04.904 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:44:04.906 DEBUG:teuthology.orchestra.run.smithi186:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-04T17:44:04.959 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-04T17:44:04.959 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-05-04T17:44:04.960 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-04T17:44:04.960 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-05-04T17:44:04.962 INFO:tasks.ceph.mon.a:Started 2024-05-04T17:44:04.963 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-05-04T17:44:04.963 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-04T17:44:04.963 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-05-04T17:44:04.965 INFO:tasks.ceph.mon.c:Started 2024-05-04T17:44:04.965 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-05-04T17:44:04.965 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-04T17:44:04.966 DEBUG:teuthology.orchestra.run.smithi186:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-05-04T17:44:05.002 INFO:tasks.ceph.mon.b:Started 2024-05-04T17:44:05.002 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-04T17:44:05.002 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-04T17:44:05.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-05-04T17:44:05.005 INFO:tasks.ceph.mgr.y:Started 2024-05-04T17:44:05.005 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-04T17:44:05.005 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-05-04T17:44:05.007 INFO:tasks.ceph.mgr.x:Started 2024-05-04T17:44:05.007 INFO:tasks.ceph:Setting crush tunables to default 2024-05-04T17:44:05.007 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd crush tunables default 2024-05-04T17:44:05.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.126+0000 7f6ac29ab640 10 monclient: get_monmap_and_config 2024-05-04T17:44:05.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.126+0000 7f6ac29ab640 10 monclient: build_initial_monmap 2024-05-04T17:44:05.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.126+0000 7f6ac29ab640 10 monclient: monmap: 2024-05-04T17:44:05.128 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:44:05.128 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:05.128 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:44:05.127568+0000 2024-05-04T17:44:05.128 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:44:05.127568+0000 2024-05-04T17:44:05.128 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:05.128 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:05.128 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:05.128 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:05.128 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:05.128 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:05.129 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:05.129 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:05.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.127+0000 7f6ac29ab640 1 Processor -- start 2024-05-04T17:44:05.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.127+0000 7f6ac29ab640 1 -- start start 2024-05-04T17:44:05.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.127+0000 7f6ac29ab640 10 monclient: init 2024-05-04T17:44:05.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac29ab640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:05.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac29ab640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-05-04T17:44:05.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac29ab640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6abc0a9fb0 addr v1:172.21.15.186:6789/0 2024-05-04T17:44:05.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac29ab640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6abc1404e0 0x7f6abc0aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:05.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac29ab640 10 monclient(hunting): picked mon.noname-a con 0x7f6abc1404e0 addr v2:172.21.15.19:3300/0 2024-05-04T17:44:05.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac29ab640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6abc0b27f0 addr v1:172.21.15.19:6789/0 2024-05-04T17:44:05.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac29ab640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6abc0ab880 con 0x7f6abc0b27f0 2024-05-04T17:44:05.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac29ab640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6abc146b00 con 0x7f6abc0a9fb0 2024-05-04T17:44:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac29ab640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac29ab640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f6abc146cd0 con 0x7f6abc1404e0 2024-05-04T17:44:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac29ab640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac29ab640 10 monclient(hunting): authenticate will time out at 1267.241889s 2024-05-04T17:44:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6abb7fe640 1 -- >> v2:172.21.15.19:3300/0 conn(0x7f6abc1404e0 msgr2=0x7f6abc0aa390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3300/0 2024-05-04T17:44:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6ac0f48640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f6abc0b27f0 legacy=0x7f6abc1463f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-05-04T17:44:05.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6abb7fe640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6abc1404e0 0x7f6abc0aa390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T17:44:05.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.128+0000 7f6abbfff640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f6abc0a9fb0 legacy=0x7f6abc13fdd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-05-04T17:44:05.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.329+0000 7f6abb7fe640 1 -- >> v2:172.21.15.19:3300/0 conn(0x7f6abc1404e0 msgr2=0x7f6abc0aa390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3300/0 2024-05-04T17:44:05.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.329+0000 7f6ac0f48640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f6abc0b27f0 legacy=0x7f6abc1463f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-05-04T17:44:05.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.329+0000 7f6abb7fe640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6abc1404e0 0x7f6abc0aa390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T17:44:05.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.329+0000 7f6abbfff640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f6abc0a9fb0 legacy=0x7f6abc13fdd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-05-04T17:44:05.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.730+0000 7f6abb7fe640 1 -- >> v2:172.21.15.19:3300/0 conn(0x7f6abc1404e0 msgr2=0x7f6abc0aa390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3300/0 2024-05-04T17:44:05.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.730+0000 7f6ac0f48640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f6abc0b27f0 legacy=0x7f6abc1463f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-05-04T17:44:05.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.730+0000 7f6abb7fe640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6abc1404e0 0x7f6abc0aa390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-04T17:44:05.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:05.730+0000 7f6abbfff640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f6abc0a9fb0 legacy=0x7f6abc13fdd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-05-04T17:44:06.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:06.531+0000 7f6abb7fe640 1 -- >> v2:172.21.15.19:3300/0 conn(0x7f6abc1404e0 msgr2=0x7f6abc0aa390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3300/0 2024-05-04T17:44:06.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:06.531+0000 7f6ac0f48640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f6abc0b27f0 legacy=0x7f6abc1463f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.19:6789/0 2024-05-04T17:44:06.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:06.531+0000 7f6abb7fe640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6abc1404e0 0x7f6abc0aa390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-04T17:44:06.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:06.531+0000 7f6abbfff640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f6abc0a9fb0 legacy=0x7f6abc13fdd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-05-04T17:44:08.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 10 monclient(hunting): tick 2024-05-04T17:44:08.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 10 monclient(hunting): _check_auth_tickets 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 1 monclient(hunting): continuing hunt 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 1 -- >> v2:172.21.15.19:3300/0 conn(0x7f6abc1404e0 msgr2=0x7f6abc0aa390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6abc1404e0 0x7f6abc0aa390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7f6abc0a9fb0 legacy=0x7f6abc13fdd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 1 -- >> v1:172.21.15.19:6789/0 conn(0x7f6abc0b27f0 legacy=0x7f6abc1463f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 10 monclient: _add_conns ranks=[4,2] 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6aa80036a0 0x7f6aa8005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f6aa80036a0 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8006110 0x7f6aa80085d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f6aa8006110 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6abc146cd0 con 0x7f6aa80036a0 2024-05-04T17:44:08.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:08.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa8008cb0 con 0x7f6aa8006110 2024-05-04T17:44:08.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6ab9ffb640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:08.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6abb7fe640 1 -- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8006110 msgr2=0x7f6aa80085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3301/0 2024-05-04T17:44:08.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6abb7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8006110 0x7f6aa80085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T17:44:08.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6abbfff640 1 -- >> v2:172.21.15.186:3300/0 conn(0x7f6aa80036a0 msgr2=0x7f6aa8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-05-04T17:44:08.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.128+0000 7f6abbfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6aa80036a0 0x7f6aa8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T17:44:08.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.329+0000 7f6abb7fe640 1 -- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8006110 msgr2=0x7f6aa80085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3301/0 2024-05-04T17:44:08.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.329+0000 7f6abb7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8006110 0x7f6aa80085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T17:44:08.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.329+0000 7f6abbfff640 1 -- >> v2:172.21.15.186:3300/0 conn(0x7f6aa80036a0 msgr2=0x7f6aa8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-05-04T17:44:08.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.329+0000 7f6abbfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6aa80036a0 0x7f6aa8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T17:44:08.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.730+0000 7f6abb7fe640 1 -- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8006110 msgr2=0x7f6aa80085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3301/0 2024-05-04T17:44:08.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.730+0000 7f6abb7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8006110 0x7f6aa80085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-04T17:44:08.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.730+0000 7f6abbfff640 1 -- >> v2:172.21.15.186:3300/0 conn(0x7f6aa80036a0 msgr2=0x7f6aa8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-05-04T17:44:08.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:08.730+0000 7f6abbfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6aa80036a0 0x7f6aa8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-04T17:44:09.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:09.531+0000 7f6abb7fe640 1 -- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8006110 msgr2=0x7f6aa80085d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3301/0 2024-05-04T17:44:09.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:09.531+0000 7f6abb7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8006110 0x7f6aa80085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-04T17:44:09.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:09.531+0000 7f6abbfff640 1 -- >> v2:172.21.15.186:3300/0 conn(0x7f6aa80036a0 msgr2=0x7f6aa8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-05-04T17:44:09.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:09.531+0000 7f6abbfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6aa80036a0 0x7f6aa8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-04T17:44:11.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 10 monclient(hunting): tick 2024-05-04T17:44:11.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 10 monclient(hunting): _check_auth_tickets 2024-05-04T17:44:11.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 1 monclient(hunting): continuing hunt 2024-05-04T17:44:11.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-05-04T17:44:11.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 1 -- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8006110 msgr2=0x7f6aa80085d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-04T17:44:11.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8006110 0x7f6aa80085d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:11.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 1 -- >> v2:172.21.15.186:3300/0 conn(0x7f6aa80036a0 msgr2=0x7f6aa8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-04T17:44:11.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6aa80036a0 0x7f6aa8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:11.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ac0f48640 1 -- reap_dead start 2024-05-04T17:44:11.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-05-04T17:44:11.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6aa8003b10 0x7f6aa80065c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:11.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f6aa8003b10 addr v2:172.21.15.19:3300/0 2024-05-04T17:44:11.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6aa8006b00 0x7f6aa8006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:11.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f6aa8006b00 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:11.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8007420 0x7f6aa80078b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:11.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f6aa8007420 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:11.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:11.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f6abc146cd0 con 0x7f6aa8003b10 2024-05-04T17:44:11.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:11.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6aa8008cb0 con 0x7f6aa8006b00 2024-05-04T17:44:11.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:11.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa80080b0 con 0x7f6aa8007420 2024-05-04T17:44:11.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ab9ffb640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:11.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6abbfff640 1 -- >> v2:172.21.15.19:3300/0 conn(0x7f6aa8003b10 msgr2=0x7f6aa80065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3300/0 2024-05-04T17:44:11.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6abbfff640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6aa8003b10 0x7f6aa80065c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T17:44:11.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ac0f48640 1 -- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8007420 msgr2=0x7f6aa80078b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3301/0 2024-05-04T17:44:11.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6abb7fe640 1 -- >> v2:172.21.15.186:3300/0 conn(0x7f6aa8006b00 msgr2=0x7f6aa8006ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-05-04T17:44:11.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6ac0f48640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8007420 0x7f6aa80078b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T17:44:11.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.128+0000 7f6abb7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6aa8006b00 0x7f6aa8006ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T17:44:11.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.329+0000 7f6abbfff640 1 -- >> v2:172.21.15.19:3300/0 conn(0x7f6aa8003b10 msgr2=0x7f6aa80065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3300/0 2024-05-04T17:44:11.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.329+0000 7f6ac0f48640 1 -- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8007420 msgr2=0x7f6aa80078b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3301/0 2024-05-04T17:44:11.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.329+0000 7f6abbfff640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6aa8003b10 0x7f6aa80065c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T17:44:11.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.329+0000 7f6abb7fe640 1 -- >> v2:172.21.15.186:3300/0 conn(0x7f6aa8006b00 msgr2=0x7f6aa8006ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-05-04T17:44:11.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.329+0000 7f6ac0f48640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8007420 0x7f6aa80078b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T17:44:11.330 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.329+0000 7f6abb7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6aa8006b00 0x7f6aa8006ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T17:44:11.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.730+0000 7f6abbfff640 1 -- >> v2:172.21.15.19:3300/0 conn(0x7f6aa8003b10 msgr2=0x7f6aa80065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3300/0 2024-05-04T17:44:11.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.730+0000 7f6ac0f48640 1 -- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8007420 msgr2=0x7f6aa80078b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3301/0 2024-05-04T17:44:11.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.730+0000 7f6abbfff640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6aa8003b10 0x7f6aa80065c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-04T17:44:11.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.730+0000 7f6ac0f48640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8007420 0x7f6aa80078b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-04T17:44:11.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.730+0000 7f6abb7fe640 1 -- >> v2:172.21.15.186:3300/0 conn(0x7f6aa8006b00 msgr2=0x7f6aa8006ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-05-04T17:44:11.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:11.730+0000 7f6abb7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6aa8006b00 0x7f6aa8006ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-04T17:44:11.965 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:00:06.854 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:44:12.273 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:00:07.162 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:44:12.416 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:00:07.346 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:44:12.419 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:00:07.350 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:44:12.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:12.531+0000 7f6abbfff640 1 -- >> v2:172.21.15.19:3300/0 conn(0x7f6aa8003b10 msgr2=0x7f6aa80065c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3300/0 2024-05-04T17:44:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:12.531+0000 7f6ac0f48640 1 -- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8007420 msgr2=0x7f6aa80078b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.19:3301/0 2024-05-04T17:44:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:12.531+0000 7f6abbfff640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6aa8003b10 0x7f6aa80065c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-04T17:44:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:12.531+0000 7f6ac0f48640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8007420 0x7f6aa80078b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-04T17:44:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:12.531+0000 7f6abb7fe640 1 -- >> v2:172.21.15.186:3300/0 conn(0x7f6aa8006b00 msgr2=0x7f6aa8006ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-05-04T17:44:12.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:12.531+0000 7f6abb7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6aa8006b00 0x7f6aa8006ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-04T17:44:12.743 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:00:07.674 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:44:12.746 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:00:07.677 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:44:14.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 10 monclient(hunting): tick 2024-05-04T17:44:14.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 10 monclient(hunting): _check_auth_tickets 2024-05-04T17:44:14.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 1 monclient(hunting): continuing hunt 2024-05-04T17:44:14.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-05-04T17:44:14.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 1 -- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8007420 msgr2=0x7f6aa80078b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-04T17:44:14.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8007420 0x7f6aa80078b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:14.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 1 -- >> v2:172.21.15.186:3300/0 conn(0x7f6aa8006b00 msgr2=0x7f6aa8006ee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-04T17:44:14.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6aa8006b00 0x7f6aa8006ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:14.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 1 -- >> v2:172.21.15.19:3300/0 conn(0x7f6aa8003b10 msgr2=0x7f6aa80065c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-04T17:44:14.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6aa8003b10 0x7f6aa80065c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:14.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 10 monclient: _add_conns ranks=[1,0] 2024-05-04T17:44:14.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6aa80083b0 addr v1:172.21.15.186:6789/0 2024-05-04T17:44:14.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6aa8010720 addr v1:172.21.15.19:6789/0 2024-05-04T17:44:14.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6abc146e50 con 0x7f6aa8010720 2024-05-04T17:44:14.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6abc0ab880 con 0x7f6aa80083b0 2024-05-04T17:44:14.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.128+0000 7f6ab9ffb640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:14.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.141+0000 7f6abbfff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6aa80083b0 0x7f6aa800f000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:54516/0 (socket says 172.21.15.19:54516) 2024-05-04T17:44:14.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.141+0000 7f6abbfff640 1 -- 172.21.15.19:0/2405532677 learned_addr learned my addr 172.21.15.19:0/2405532677 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:44:14.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.629+0000 7f6abb7fe640 1 -- 172.21.15.19:0/2405532677 >> v1:172.21.15.19:6789/0 conn(0x7f6aa8010720 legacy=0x7f6aa8012b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-04T17:44:14.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.629+0000 7f6abb7fe640 1 -- 172.21.15.19:0/2405532677 >> v1:172.21.15.19:6789/0 conn(0x7f6aa8010720 legacy=0x7f6aa8012b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-04T17:44:14.630 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:14.629+0000 7f6abb7fe640 1 --1- 172.21.15.19:0/2405532677 >> v1:172.21.15.19:6789/0 conn(0x7f6aa8010720 0x7f6aa8012b10 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-05-04T17:44:17.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.128+0000 7f6ab9ffb640 10 monclient(hunting): tick 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.128+0000 7f6ab9ffb640 10 monclient(hunting): _check_auth_tickets 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.128+0000 7f6ab9ffb640 1 monclient(hunting): continuing hunt 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.128+0000 7f6ab9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.128+0000 7f6ab9ffb640 1 -- 172.21.15.19:0/2405532677 >> v1:172.21.15.186:6789/0 conn(0x7f6aa80083b0 legacy=0x7f6aa800f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.128+0000 7f6ab9ffb640 1 -- 172.21.15.19:0/2405532677 >> v1:172.21.15.19:6789/0 conn(0x7f6aa8010720 legacy=0x7f6aa8012b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ab9ffb640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ac0f48640 1 -- 172.21.15.19:0/2405532677 reap_dead start 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ab9ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6aa8014250 addr v1:172.21.15.186:6789/0 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ab9ffb640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 0x7f6aa80078b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ab9ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f6aa8015e70 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ab9ffb640 1 --2- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 0x7f6aa80065a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ab9ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f6aa8005b00 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:17.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ab9ffb640 1 -- 172.21.15.19:0/2405532677 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6abc146b00 con 0x7f6aa8014250 2024-05-04T17:44:17.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ab9ffb640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:17.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ab9ffb640 1 -- 172.21.15.19:0/2405532677 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6abc146cd0 con 0x7f6aa8005b00 2024-05-04T17:44:17.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ab9ffb640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:17.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ab9ffb640 1 -- 172.21.15.19:0/2405532677 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa8008cb0 con 0x7f6aa8015e70 2024-05-04T17:44:17.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6ab9ffb640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:17.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.129+0000 7f6abb7fe640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 0x7f6aa80078b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:17.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.130+0000 7f6abb7fe640 10 monclient(hunting): get_auth_request con 0x7f6aa8015e70 auth_method 0 2024-05-04T17:44:17.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.130+0000 7f6abb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:17.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.130+0000 7f6abb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:17.132 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.130+0000 7f6abb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:17.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.130+0000 7f6abb7fe640 1 -- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 msgr2=0x7f6aa80078b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-04T17:44:17.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.130+0000 7f6abb7fe640 1 -- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 msgr2=0x7f6aa80078b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-04T17:44:17.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.130+0000 7f6abb7fe640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 0x7f6aa80078b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-04T17:44:17.133 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.131+0000 7f6abb7fe640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 0x7f6aa80078b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T17:44:17.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.143+0000 7f6ac0f48640 1 --2- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 0x7f6aa80065a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:17.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.151+0000 7f6ac0f48640 10 monclient(hunting): get_auth_request con 0x7f6aa8005b00 auth_method 0 2024-05-04T17:44:17.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.151+0000 7f6ac0f48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:17.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.151+0000 7f6ac0f48640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:17.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.151+0000 7f6ac0f48640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:17.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.162+0000 7f6ac0f48640 1 -- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 msgr2=0x7f6aa80065a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-04T17:44:17.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.162+0000 7f6ac0f48640 1 -- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 msgr2=0x7f6aa80065a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-04T17:44:17.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.162+0000 7f6ac0f48640 1 --2- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 0x7f6aa80065a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-04T17:44:17.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.162+0000 7f6ac0f48640 1 --2- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 0x7f6aa80065a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T17:44:17.466 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.465+0000 7f6ac0f48640 1 --2- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 0x7f6aa80065a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:17.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.466+0000 7f6ac0f48640 10 monclient(hunting): get_auth_request con 0x7f6aa8005b00 auth_method 0 2024-05-04T17:44:17.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.466+0000 7f6ac0f48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:17.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.466+0000 7f6ac0f48640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:17.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.466+0000 7f6ac0f48640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-04T17:44:17.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.480+0000 7f6abb7fe640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 0x7f6aa80078b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:17.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.482+0000 7f6abb7fe640 10 monclient(hunting): get_auth_request con 0x7f6aa8015e70 auth_method 0 2024-05-04T17:44:17.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.482+0000 7f6abb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:17.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.482+0000 7f6abb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:17.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.482+0000 7f6abb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-04T17:44:17.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.499+0000 7f6ac0f48640 1 -- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 msgr2=0x7f6aa80065a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-04T17:44:17.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.499+0000 7f6ac0f48640 1 -- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 msgr2=0x7f6aa80065a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-04T17:44:17.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.499+0000 7f6ac0f48640 1 --2- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 0x7f6aa80065a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-04T17:44:17.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.499+0000 7f6ac0f48640 1 --2- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 0x7f6aa80065a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T17:44:17.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.513+0000 7f6abb7fe640 1 -- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 msgr2=0x7f6aa80078b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-04T17:44:17.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.513+0000 7f6abb7fe640 1 -- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 msgr2=0x7f6aa80078b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-04T17:44:17.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.513+0000 7f6abb7fe640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 0x7f6aa80078b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-04T17:44:17.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.513+0000 7f6abb7fe640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 0x7f6aa80078b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-04T17:44:17.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.831+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 402493903 0 0) 0x7f6abc146b00 con 0x7f6aa8014250 2024-05-04T17:44:17.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.831+0000 7f6abaffd640 10 monclient(hunting): _negotiate 2024-05-04T17:44:17.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.831+0000 7f6abaffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:17.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.831+0000 7f6abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:17.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.831+0000 7f6abaffd640 10 monclient(hunting): my global_id is 4105 2024-05-04T17:44:17.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.832+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6aa4003520 con 0x7f6aa8014250 2024-05-04T17:44:17.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.917+0000 7f6ac0f48640 1 --2- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 0x7f6aa80065a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:17.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.921+0000 7f6ac0f48640 10 monclient(hunting): get_auth_request con 0x7f6aa8005b00 auth_method 0 2024-05-04T17:44:17.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.921+0000 7f6ac0f48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:17.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.921+0000 7f6ac0f48640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:17.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.921+0000 7f6ac0f48640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-04T17:44:17.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.927+0000 7f6ac0f48640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:17.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.927+0000 7f6ac0f48640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:17.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.927+0000 7f6ac0f48640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:17.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3977204437 0 0) 0x7f6aa4003520 con 0x7f6aa8014250 2024-05-04T17:44:17.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 msgr2=0x7f6aa80078b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 0x7f6aa80078b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 msgr2=0x7f6aa80065a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 --2- 172.21.15.19:0/2405532677 >> v2:172.21.15.186:3300/0 conn(0x7f6aa8005b00 0x7f6aa80065a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient: _finish_hunting 0 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 monclient: found mon.noname-c-legacy 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.186:6789/0 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa8013ba0 con 0x7f6aa8014250 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient: _finish_auth 0 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient: _check_auth_tickets 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3626739996 0 0) 0x7f6aac0040b0 con 0x7f6aa8014250 2024-05-04T17:44:17.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient: dump: 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] but i'm connected to v1:172.21.15.186:6789/0, reconnecting 2024-05-04T17:44:17.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 >> v1:172.21.15.186:6789/0 conn(0x7f6aa8014250 legacy=0x7f6aa8014690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6aa40039b0 0x7f6aa4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient(hunting): picked mon.a con 0x7f6aa40039b0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 0x7f6aa4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient(hunting): picked mon.c con 0x7f6aa8015e70 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6aa4004d40 0x7f6aa40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient(hunting): picked mon.b con 0x7f6aa4004d40 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6abc146cd0 con 0x7f6aa40039b0 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa8008cb0 con 0x7f6aa4004d40 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa400ba40 con 0x7f6aa8015e70 2024-05-04T17:44:17.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:17.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abaffd640 10 monclient(hunting): _finish_auth 0 2024-05-04T17:44:17.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.960+0000 7f6abbfff640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6aa40039b0 0x7f6aa4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:17.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.961+0000 7f6abbfff640 10 monclient(hunting): get_auth_request con 0x7f6aa40039b0 auth_method 0 2024-05-04T17:44:17.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.961+0000 7f6abbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:17.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.961+0000 7f6abbfff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:17.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.961+0000 7f6abbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-04T17:44:17.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.962+0000 7f6abbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:17.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.962+0000 7f6abbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:17.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.962+0000 7f6abbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-04T17:44:17.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.972+0000 7f6abb7fe640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 0x7f6aa4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:17.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.975+0000 7f6abb7fe640 10 monclient(hunting): get_auth_request con 0x7f6aa8015e70 auth_method 0 2024-05-04T17:44:17.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.975+0000 7f6abb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:17.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.975+0000 7f6abb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:17.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.975+0000 7f6abb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-04T17:44:17.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:17.996+0000 7f6ac0f48640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6aa4004d40 0x7f6aa40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:18.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.000+0000 7f6ac0f48640 10 monclient(hunting): get_auth_request con 0x7f6aa4004d40 auth_method 0 2024-05-04T17:44:18.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.000+0000 7f6ac0f48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:18.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.000+0000 7f6ac0f48640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:18.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.000+0000 7f6ac0f48640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-04T17:44:18.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.004+0000 7f6ac0f48640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:18.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.004+0000 7f6ac0f48640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:18.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.004+0000 7f6ac0f48640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-04T17:44:18.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.010+0000 7f6abb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:18.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.010+0000 7f6abb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:18.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.010+0000 7f6abb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-04T17:44:18.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 10 monclient(hunting): handle_auth_done global_id 4105 payload 293 2024-05-04T17:44:18.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 1 -- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 msgr2=0x7f6aa4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:18.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6aa8015e70 0x7f6aa4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:18.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 1 -- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6aa40039b0 msgr2=0x7f6aa4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:18.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6aa40039b0 0x7f6aa4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:18.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 10 monclient: _finish_hunting 0 2024-05-04T17:44:18.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 1 monclient: found mon.b 2024-05-04T17:44:18.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:18.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:18.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 1 -- 172.21.15.19:0/2405532677 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa4003520 con 0x7f6aa4004d40 2024-05-04T17:44:18.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 10 monclient: _finish_auth 0 2024-05-04T17:44:18.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 10 monclient: _check_auth_tickets 2024-05-04T17:44:18.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac0f48640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:18.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac29ab640 5 monclient: authenticate success, global_id 4105 2024-05-04T17:44:18.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.012+0000 7f6ac29ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:44:18.028 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.028+0000 7f6ac0f48640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6aa4004d40 0x7f6aa40051c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0b4820 tx=0x7f6aa4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba7dcfd049b89041 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:18.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.035+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4009c70 con 0x7f6aa4004d40 2024-05-04T17:44:18.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.035+0000 7f6abaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:18.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.035+0000 7f6abaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:18.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.035+0000 7f6abaffd640 10 monclient: dump: 2024-05-04T17:44:18.036 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:44:18.036 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:18.036 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:18.036 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:18.036 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:18.036 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.035+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ab4039070 con 0x7f6aa4004d40 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.035+0000 7f6abaffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.035+0000 7f6abaffd640 1 -- 172.21.15.19:0/2405532677 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab40337f0 con 0x7f6aa4004d40 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.035+0000 7f6ac29ab640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.035+0000 7f6ac29ab640 4 set_mon_vals no callback set 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.035+0000 7f6abaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.035+0000 7f6abaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.035+0000 7f6abaffd640 10 monclient: dump: 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:18.037 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:18.038 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:18.038 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:18.038 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:18.038 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:18.038 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:18.038 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:18.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.036+0000 7f6ac29ab640 10 monclient: shutdown 2024-05-04T17:44:18.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.036+0000 7f6ac29ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:18.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.036+0000 7f6ac29ab640 1 -- 172.21.15.19:0/2405532677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6aa4004d40 msgr2=0x7f6aa40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:18.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.036+0000 7f6ac29ab640 1 --2- 172.21.15.19:0/2405532677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6aa4004d40 0x7f6aa40051c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0b4820 tx=0x7f6aa4005830 comp rx=0 tx=0).stop 2024-05-04T17:44:18.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.036+0000 7f6ac0f48640 1 -- 172.21.15.19:0/2405532677 reap_dead start 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.036+0000 7f6ac29ab640 1 -- 172.21.15.19:0/2405532677 shutdown_connections 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.036+0000 7f6ac29ab640 1 -- 172.21.15.19:0/2405532677 >> 172.21.15.19:0/2405532677 conn(0x7f6abc0b4e90 msgr2=0x7f6abc0b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.036+0000 7f6ac29ab640 1 -- 172.21.15.19:0/2405532677 shutdown_connections 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.036+0000 7f6ac29ab640 1 -- 172.21.15.19:0/2405532677 wait complete. 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.037+0000 7f6ac29ab640 10 monclient: build_initial_monmap 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.037+0000 7f6ac29ab640 10 monclient: monmap: 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:18.041 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.037+0000 7f6ac29ab640 1 librados: starting msgr at 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.037+0000 7f6ac29ab640 1 librados: starting objecter 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.037+0000 7f6ac29ab640 1 Processor -- start 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.037+0000 7f6ac29ab640 1 -- start start 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.037+0000 7f6ac29ab640 1 librados: setting wanted keys 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.037+0000 7f6ac29ab640 1 librados: calling monclient init 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.037+0000 7f6ac29ab640 10 monclient: init 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6abc141520 0x7f6abc143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 10 monclient(hunting): picked mon.noname-a con 0x7f6abc141520 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:18.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6abc141c40 0x7f6abc144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:18.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 10 monclient(hunting): picked mon.noname-c con 0x7f6abc141c40 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:18.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6abc136860 0x7f6abc136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:18.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 10 monclient(hunting): picked mon.noname-b con 0x7f6abc136860 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:18.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:18.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6abc0aa720 con 0x7f6abc141520 2024-05-04T17:44:18.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:18.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6abc0aa5a0 con 0x7f6abc136860 2024-05-04T17:44:18.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:18.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f6abc13fdd0 con 0x7f6abc141c40 2024-05-04T17:44:18.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:18.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.038+0000 7f6ac29ab640 10 monclient(hunting): authenticate will time out at 1280.152040s 2024-05-04T17:44:18.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.046+0000 7f6abbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6abc141520 0x7f6abc143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:18.047 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.047+0000 7f6abbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6abc141520 0x7f6abc143ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:52752/0 (socket says 172.21.15.19:52752) 2024-05-04T17:44:18.047 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.047+0000 7f6abbfff640 1 -- 172.21.15.19:0/1931044306 learned_addr learned my addr 172.21.15.19:0/1931044306 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:44:18.047 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.047+0000 7f6abbfff640 10 monclient(hunting): get_auth_request con 0x7f6abc141520 auth_method 0 2024-05-04T17:44:18.048 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.047+0000 7f6abbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:18.048 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.047+0000 7f6abbfff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:18.048 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.047+0000 7f6abbfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:18.049 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.048+0000 7f6abb7fe640 1 --2- 172.21.15.19:0/1931044306 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6abc141c40 0x7f6abc144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:18.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.049+0000 7f6abb7fe640 10 monclient(hunting): get_auth_request con 0x7f6abc141c40 auth_method 0 2024-05-04T17:44:18.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.049+0000 7f6abb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:18.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.049+0000 7f6abb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:18.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.049+0000 7f6abb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:18.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.050+0000 7f6abb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:18.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.050+0000 7f6abb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:18.051 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.050+0000 7f6abb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:18.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.061+0000 7f6abb7fe640 10 monclient(hunting): handle_auth_done global_id 4103 payload 824 2024-05-04T17:44:18.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.061+0000 7f6abb7fe640 1 -- 172.21.15.19:0/1931044306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6abc136860 msgr2=0x7f6abc136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:18.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.061+0000 7f6abb7fe640 1 --2- 172.21.15.19:0/1931044306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6abc136860 0x7f6abc136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:18.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.061+0000 7f6abb7fe640 1 -- 172.21.15.19:0/1931044306 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6abc141520 msgr2=0x7f6abc143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:18.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.061+0000 7f6abb7fe640 1 --2- 172.21.15.19:0/1931044306 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6abc141520 0x7f6abc143ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:18.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.061+0000 7f6abb7fe640 10 monclient: _finish_hunting 0 2024-05-04T17:44:18.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.061+0000 7f6abb7fe640 1 monclient: found mon.noname-c 2024-05-04T17:44:18.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.061+0000 7f6abb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:18.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.061+0000 7f6abb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T17:44:18.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.061+0000 7f6abb7fe640 1 -- 172.21.15.19:0/1931044306 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6abc143630 con 0x7f6abc141c40 2024-05-04T17:44:18.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.064+0000 7f6abb7fe640 1 --2- 172.21.15.19:0/1931044306 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6abc141c40 0x7f6abc144530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000ce50 tx=0x7f6ab000b780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:18.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.064+0000 7f6ab8ff9640 10 client.?.objecter ms_handle_connect 0x7f6abc141c40 2024-05-04T17:44:18.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.064+0000 7f6ab8ff9640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:18.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 1 -- 172.21.15.19:0/1931044306 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0018630 con 0x7f6abc141c40 2024-05-04T17:44:18.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:18.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 10 monclient: dump: 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 10 monclient: _finish_auth 0 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 10 monclient: _check_auth_tickets 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 1 -- 172.21.15.19:0/1931044306 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ab00187d0 con 0x7f6abc141c40 2024-05-04T17:44:18.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:18.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ac29ab640 5 monclient: authenticate success, global_id 4103 2024-05-04T17:44:18.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 1 -- 172.21.15.19:0/1931044306 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0018ba0 con 0x7f6abc141c40 2024-05-04T17:44:18.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:18.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:44:18.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ab8ff9640 10 monclient: dump: 2024-05-04T17:44:18.072 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:44:18.072 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:18.072 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:18.072 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:18.072 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ac29ab640 10 monclient: _renew_subs 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ac29ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6aba7fc640 4 set_mon_vals no callback set 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ac29ab640 1 -- 172.21.15.19:0/1931044306 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6abc1421d0 con 0x7f6abc141c40 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ac29ab640 10 client.4103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ac29ab640 10 monclient: _renew_subs 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ac29ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ac29ab640 1 -- 172.21.15.19:0/1931044306 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6abc143910 con 0x7f6abc141c40 2024-05-04T17:44:18.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.069+0000 7f6ac29ab640 1 librados: init done 2024-05-04T17:44:18.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.070+0000 7f6ac29ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:18.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.070+0000 7f6ac29ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:18.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.070+0000 7f6ac29ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:44:18.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.070+0000 7f6ac29ab640 1 -- 172.21.15.19:0/1931044306 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6abc068b10 con 0x7f6abc141c40 2024-05-04T17:44:18.097 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.096+0000 7f6ab8ff9640 1 -- 172.21.15.19:0/1931044306 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6ab0009020 con 0x7f6abc141c40 2024-05-04T17:44:18.097 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.096+0000 7f6ab8ff9640 1 -- 172.21.15.19:0/1931044306 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6ab00173e0 con 0x7f6abc141c40 2024-05-04T17:44:18.097 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.096+0000 7f6ab8ff9640 10 client.4103.objecter ms_dispatch 0x7f6abc000b90 osd_map(1..1 src has 1..1) 2024-05-04T17:44:18.097 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.096+0000 7f6ab8ff9640 3 client.4103.objecter handle_osd_map got epochs [1,1] > 0 2024-05-04T17:44:18.097 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.096+0000 7f6ab8ff9640 3 client.4103.objecter handle_osd_map decoding full epoch 1 2024-05-04T17:44:18.097 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.096+0000 7f6ab8ff9640 20 client.4103.objecter dump_active .. 0 homeless 2024-05-04T17:44:18.186 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:18.185+0000 7f9ef103e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-04T17:44:18.270 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:18.269+0000 7f33478fa040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-04T17:44:18.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.300+0000 7f6ab8ff9640 1 -- 172.21.15.19:0/1931044306 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6ab0026070 con 0x7f6abc141c40 2024-05-04T17:44:18.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.300+0000 7f6ab8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:18.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.300+0000 7f6ab8ff9640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:18.357 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:18.356+0000 7f9ef103e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-04T17:44:18.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.375+0000 7f6ac29ab640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-05-04T17:44:18.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.375+0000 7f6ac29ab640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-05-04T17:44:18.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.375+0000 7f6ac29ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:44:18.376 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.375+0000 7f6ac29ab640 1 -- 172.21.15.19:0/1931044306 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f6abc13d550 con 0x7f6abc141c40 2024-05-04T17:44:18.415 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:18.415+0000 7f9ef103e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-04T17:44:18.434 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:18.433+0000 7f33478fa040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-04T17:44:18.487 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:18.486+0000 7f33478fa040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-04T17:44:18.673 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:18.672+0000 7f9ef103e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-04T17:44:18.727 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:18.726+0000 7f9ef103e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-04T17:44:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.750+0000 7f6ab8ff9640 1 -- 172.21.15.19:0/1931044306 <== mon.2 v2:172.21.15.19:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f6abc143630 con 0x7f6abc141c40 2024-05-04T17:44:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.750+0000 7f6ab8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-05-04T17:44:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.750+0000 7f6ab8ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-05-04T17:44:18.751 INFO:teuthology.orchestra.run.smithi019.stderr:adjusted tunables profile to default 2024-05-04T17:44:18.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 10 librados: watch_flush enter 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 10 librados: watch_flush exit 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 20 client.4103.objecter shutdown clearing up homeless session... 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 10 client.4103.objecter successfully canceled tick 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 10 monclient: shutdown 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 1 -- 172.21.15.19:0/1931044306 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6abc141c40 msgr2=0x7f6abc144530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 1 --2- 172.21.15.19:0/1931044306 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6abc141c40 0x7f6abc144530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000ce50 tx=0x7f6ab000b780 comp rx=0 tx=0).stop 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 1 -- 172.21.15.19:0/1931044306 shutdown_connections 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 1 --2- 172.21.15.19:0/1931044306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6abc136860 0x7f6abc136ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 1 --2- 172.21.15.19:0/1931044306 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6abc141c40 0x7f6abc144530 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 1 --2- 172.21.15.19:0/1931044306 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6abc141520 0x7f6abc143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 1 -- 172.21.15.19:0/1931044306 >> 172.21.15.19:0/1931044306 conn(0x7f6abc0b4e90 msgr2=0x7f6abc13cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:18.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.751+0000 7f6ac29ab640 1 -- 172.21.15.19:0/1931044306 shutdown_connections 2024-05-04T17:44:18.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.752+0000 7f6ac29ab640 1 -- 172.21.15.19:0/1931044306 wait complete. 2024-05-04T17:44:18.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:18.752+0000 7f6ac29ab640 1 librados: shutdown 2024-05-04T17:44:18.754 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:18.753+0000 7f33478fa040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-04T17:44:18.764 INFO:tasks.ceph:check_enable_crimson: False 2024-05-04T17:44:18.764 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-04T17:44:18.765 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:44:18.765 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-04T17:44:18.791 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:44:18.791 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-04T17:44:18.801 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:18.800+0000 7f33478fa040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-04T17:44:18.859 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:44:18.859 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-04T17:44:18.925 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:44:18.925 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-04T17:44:18.989 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:44:18.989 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-04T17:44:19.013 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:44:19.013 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-04T17:44:19.076 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:44:19.077 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-04T17:44:19.140 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-04T17:44:19.140 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-04T17:44:19.204 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 4b749ec0-22b6-458c-a7af-2ccace806ab7 0 2024-05-04T17:44:19.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.340+0000 7fa675504640 10 monclient: get_monmap_and_config 2024-05-04T17:44:19.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.340+0000 7fa675504640 10 monclient: build_initial_monmap 2024-05-04T17:44:19.341 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.340+0000 7fa675504640 10 monclient: monmap: 2024-05-04T17:44:19.341 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:19.341 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:19.341 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:44:19.341185+0000 2024-05-04T17:44:19.341 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:44:19.341185+0000 2024-05-04T17:44:19.341 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:19.341 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:19.341 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:19.341 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:19.342 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:19.342 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:19.342 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:19.342 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:19.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.340+0000 7fa675504640 1 Processor -- start 2024-05-04T17:44:19.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 1 -- start start 2024-05-04T17:44:19.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 10 monclient: init 2024-05-04T17:44:19.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:19.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-05-04T17:44:19.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6700a9720 addr v1:172.21.15.186:6789/0 2024-05-04T17:44:19.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa6700a9b00 0x7fa6700aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:19.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 10 monclient(hunting): picked mon.noname-c con 0x7fa6700a9b00 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:19.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa6700b2e90 0x7fa67013fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:19.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 10 monclient(hunting): picked mon.noname-b con 0x7fa6700b2e90 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:19.344 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6700abc40 con 0x7fa6700a9720 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa670146fa0 con 0x7fa6700a9b00 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fa670146090 con 0x7fa6700b2e90 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.341+0000 7fa675504640 10 monclient(hunting): authenticate will time out at 1255.619235s 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.342+0000 7fa667fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa6700a9b00 0x7fa6700aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.342+0000 7fa66f7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa6700b2e90 0x7fa67013fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.342+0000 7fa667fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa6700a9b00 0x7fa6700aa890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52686/0 (socket says 172.21.15.186:52686) 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.342+0000 7fa667fff640 1 -- 172.21.15.186:0/1445164355 learned_addr learned my addr 172.21.15.186:0/1445164355 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.342+0000 7fa667fff640 10 monclient(hunting): get_auth_request con 0x7fa6700a9b00 auth_method 0 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.342+0000 7fa667fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.342+0000 7fa667fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.342+0000 7fa667fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.342+0000 7fa66f7fe640 10 monclient(hunting): get_auth_request con 0x7fa6700b2e90 auth_method 0 2024-05-04T17:44:19.345 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.342+0000 7fa66f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:19.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.342+0000 7fa66f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:19.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.342+0000 7fa66f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:19.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.343+0000 7fa667fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:19.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.343+0000 7fa667fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:19.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.343+0000 7fa667fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:19.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.343+0000 7fa66f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:19.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.343+0000 7fa66f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:19.346 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.343+0000 7fa66f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:19.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.344+0000 7fa667fff640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-05-04T17:44:19.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.344+0000 7fa667fff640 1 -- 172.21.15.186:0/1445164355 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa6700b2e90 msgr2=0x7fa67013fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:19.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.344+0000 7fa667fff640 1 --2- 172.21.15.186:0/1445164355 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa6700b2e90 0x7fa67013fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:19.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.344+0000 7fa667fff640 1 -- 172.21.15.186:0/1445164355 >> v1:172.21.15.186:6789/0 conn(0x7fa6700a9720 legacy=0x7fa670146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:19.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.344+0000 7fa667fff640 10 monclient: _finish_hunting 0 2024-05-04T17:44:19.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.344+0000 7fa667fff640 1 monclient: found mon.noname-c 2024-05-04T17:44:19.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.344+0000 7fa667fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:19.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.344+0000 7fa667fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:44:19.347 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.344+0000 7fa667fff640 1 -- 172.21.15.186:0/1445164355 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa670146370 con 0x7fa6700a9b00 2024-05-04T17:44:19.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.347+0000 7fa667fff640 1 -- 172.21.15.186:0/1445164355 >> v2:172.21.15.186:3300/0 conn(0x7fa6700a9b00 msgr2=0x7fa6700aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-04T17:44:19.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.347+0000 7fa667fff640 1 -- 172.21.15.186:0/1445164355 >> v2:172.21.15.186:3300/0 conn(0x7fa6700a9b00 msgr2=0x7fa6700aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-04T17:44:19.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.347+0000 7fa667fff640 1 --2- 172.21.15.186:0/1445164355 >> v2:172.21.15.186:3300/0 conn(0x7fa6700a9b00 0x7fa6700aa890 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7fa658009870 tx=0x7fa65802b8f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-04T17:44:19.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.347+0000 7fa667fff640 1 --2- 172.21.15.186:0/1445164355 >> v2:172.21.15.186:3300/0 conn(0x7fa6700a9b00 0x7fa6700aa890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T17:44:19.497 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:19.496+0000 7f9ef103e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-04T17:44:19.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.548+0000 7fa667fff640 1 --2- 172.21.15.186:0/1445164355 >> v2:172.21.15.186:3300/0 conn(0x7fa6700a9b00 0x7fa6700aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:19.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa667fff640 10 monclient: get_auth_request con 0x7fa6700a9b00 auth_method 0 2024-05-04T17:44:19.550 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa667fff640 0 --2- 172.21.15.186:0/1445164355 >> v2:172.21.15.186:3300/0 conn(0x7fa6700a9b00 0x7fa6700aa890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa667fff640 1 --2- 172.21.15.186:0/1445164355 >> v2:172.21.15.186:3300/0 conn(0x7fa6700a9b00 0x7fa6700aa890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 10 monclient: ms_handle_reset current mon v2:172.21.15.186:3300/0 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 1 -- 172.21.15.186:0/1445164355 >> v2:172.21.15.186:3300/0 conn(0x7fa6700a9b00 msgr2=0x7fa6700aa890 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 1 --2- 172.21.15.186:0/1445164355 >> v2:172.21.15.186:3300/0 conn(0x7fa6700a9b00 0x7fa6700aa890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 10 monclient: _add_conns ranks=[3,0] 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 1 --2- 172.21.15.186:0/1445164355 >> v2:172.21.15.19:3300/0 conn(0x7fa65c003500 0x7fa65c005980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fa65c003500 addr v2:172.21.15.19:3300/0 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa65c0062b0 addr v1:172.21.15.19:6789/0 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 1 -- 172.21.15.186:0/1445164355 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa65c00a260 con 0x7fa65c0062b0 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 1 -- 172.21.15.186:0/1445164355 --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fa65c00b4a0 con 0x7fa65c003500 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.549+0000 7fa66e7fc640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.186:3300/0 2024-05-04T17:44:19.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.550+0000 7fa66effd640 1 --2- 172.21.15.186:0/1445164355 >> v2:172.21.15.19:3300/0 conn(0x7fa65c003500 0x7fa65c005980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:19.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.550+0000 7fa66effd640 10 monclient(hunting): get_auth_request con 0x7fa65c003500 auth_method 0 2024-05-04T17:44:19.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.550+0000 7fa66effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:19.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.550+0000 7fa66effd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:19.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.550+0000 7fa66effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-04T17:44:19.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.551+0000 7fa66effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:19.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.551+0000 7fa66effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:19.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.551+0000 7fa66effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-04T17:44:19.563 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:19.563+0000 7f33478fa040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-04T17:44:19.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.565+0000 7fa66effd640 10 monclient(hunting): handle_auth_done global_id 4120 payload 293 2024-05-04T17:44:19.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.565+0000 7fa66effd640 1 -- 172.21.15.186:0/1445164355 >> v1:172.21.15.19:6789/0 conn(0x7fa65c0062b0 legacy=0x7fa65c008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:19.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.565+0000 7fa66effd640 10 monclient: _finish_hunting 0 2024-05-04T17:44:19.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.565+0000 7fa66effd640 1 monclient: found mon.noname-a 2024-05-04T17:44:19.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.565+0000 7fa66effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:19.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.565+0000 7fa66effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:44:19.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.565+0000 7fa66effd640 1 -- 172.21.15.186:0/1445164355 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65c00b710 con 0x7fa65c003500 2024-05-04T17:44:19.585 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.585+0000 7fa66effd640 1 --2- 172.21.15.186:0/1445164355 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa65c003500 0x7fa65c005980 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa6600015a0 tx=0x7fa658009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7626680b4e59b743 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:19.590 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:19.589+0000 7f9ef103e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-04T17:44:19.624 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 1 -- 172.21.15.186:0/1445164355 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660017070 con 0x7fa65c003500 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 10 monclient: dump: 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:19.625 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 10 monclient: _finish_auth 0 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 10 monclient: _check_auth_tickets 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 1 -- 172.21.15.186:0/1445164355 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6600050d0 con 0x7fa65c003500 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa675504640 5 monclient: authenticate success, global_id 4120 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa675504640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 1 -- 172.21.15.186:0/1445164355 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660001220 con 0x7fa65c003500 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa66e7fc640 10 monclient: dump: 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:19.626 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:19.627 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:19.627 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:19.627 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:19.627 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:19.627 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:19.627 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:19.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa675504640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:19.627 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.623+0000 7fa675504640 4 set_mon_vals no callback set 2024-05-04T17:44:19.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.624+0000 7fa675504640 10 monclient: shutdown 2024-05-04T17:44:19.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.624+0000 7fa675504640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:19.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.624+0000 7fa675504640 1 -- 172.21.15.186:0/1445164355 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa65c003500 msgr2=0x7fa65c005980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:19.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.624+0000 7fa675504640 1 --2- 172.21.15.186:0/1445164355 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa65c003500 0x7fa65c005980 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa6600015a0 tx=0x7fa658009240 comp rx=0 tx=0).stop 2024-05-04T17:44:19.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.624+0000 7fa66f7fe640 1 -- 172.21.15.186:0/1445164355 reap_dead start 2024-05-04T17:44:19.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.624+0000 7fa675504640 1 -- 172.21.15.186:0/1445164355 shutdown_connections 2024-05-04T17:44:19.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.624+0000 7fa675504640 1 --2- 172.21.15.186:0/1445164355 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa65c003500 0x7fa65c005980 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:19.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.624+0000 7fa675504640 1 -- 172.21.15.186:0/1445164355 >> 172.21.15.186:0/1445164355 conn(0x7fa670067780 msgr2=0x7fa6700b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:19.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.624+0000 7fa675504640 1 -- 172.21.15.186:0/1445164355 shutdown_connections 2024-05-04T17:44:19.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.624+0000 7fa675504640 1 -- 172.21.15.186:0/1445164355 wait complete. 2024-05-04T17:44:19.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.625+0000 7fa675504640 10 monclient: build_initial_monmap 2024-05-04T17:44:19.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.625+0000 7fa675504640 10 monclient: monmap: 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.625+0000 7fa675504640 1 librados: starting msgr at 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.625+0000 7fa675504640 1 librados: starting objecter 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.625+0000 7fa675504640 1 Processor -- start 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.625+0000 7fa675504640 1 -- start start 2024-05-04T17:44:19.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.625+0000 7fa675504640 1 librados: setting wanted keys 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.625+0000 7fa675504640 1 librados: calling monclient init 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.625+0000 7fa675504640 10 monclient: init 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa65c003500 0x7fa6700b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 10 monclient(hunting): picked mon.noname-b con 0x7fa65c003500 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6701dbb80 0x7fa6701393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 10 monclient(hunting): picked mon.noname-a con 0x7fa6701dbb80 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa670138a50 0x7fa670132570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 10 monclient(hunting): picked mon.noname-c con 0x7fa670138a50 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa6701407b0 con 0x7fa6701dbb80 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa670140630 con 0x7fa65c003500 2024-05-04T17:44:19.634 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fa67014d420 con 0x7fa670138a50 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa675504640 10 monclient(hunting): authenticate will time out at 1255.903993s 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.626+0000 7fa66effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa65c003500 0x7fa6700b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66f7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa670138a50 0x7fa670132570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa65c003500 0x7fa6700b7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52706/0 (socket says 172.21.15.186:52706) 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66effd640 1 -- 172.21.15.186:0/1756075573 learned_addr learned my addr 172.21.15.186:0/1756075573 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66effd640 10 monclient(hunting): get_auth_request con 0x7fa65c003500 auth_method 0 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66effd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa667fff640 1 --2- 172.21.15.186:0/1756075573 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6701dbb80 0x7fa6701393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66f7fe640 10 monclient(hunting): get_auth_request con 0x7fa670138a50 auth_method 0 2024-05-04T17:44:19.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa66effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa667fff640 10 monclient(hunting): get_auth_request con 0x7fa6701dbb80 auth_method 0 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa667fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa667fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.627+0000 7fa667fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.628+0000 7fa66f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.628+0000 7fa66f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.628+0000 7fa66f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.628+0000 7fa667fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.628+0000 7fa667fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:19.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.628+0000 7fa667fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:19.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.630+0000 7fa66effd640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-05-04T17:44:19.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.630+0000 7fa66effd640 1 -- 172.21.15.186:0/1756075573 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa670138a50 msgr2=0x7fa670132570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:19.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.630+0000 7fa66effd640 1 --2- 172.21.15.186:0/1756075573 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa670138a50 0x7fa670132570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:19.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.630+0000 7fa66effd640 1 -- 172.21.15.186:0/1756075573 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6701dbb80 msgr2=0x7fa6701393d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:19.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.630+0000 7fa66effd640 1 --2- 172.21.15.186:0/1756075573 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6701dbb80 0x7fa6701393d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:19.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.630+0000 7fa66effd640 10 monclient: _finish_hunting 0 2024-05-04T17:44:19.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.630+0000 7fa66effd640 1 monclient: found mon.noname-b 2024-05-04T17:44:19.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.630+0000 7fa66effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:19.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.630+0000 7fa66effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:19.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.630+0000 7fa66effd640 1 -- 172.21.15.186:0/1756075573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa670140950 con 0x7fa65c003500 2024-05-04T17:44:19.638 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:19.636+0000 7f9ef103e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-04T17:44:19.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.632+0000 7fa66effd640 1 --2- 172.21.15.186:0/1756075573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa65c003500 0x7fa6700b7540 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa660001390 tx=0x7fa660000fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:19.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.632+0000 7fa6677fe640 10 client.?.objecter ms_handle_connect 0x7fa65c003500 2024-05-04T17:44:19.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.632+0000 7fa6677fe640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:19.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 1 -- 172.21.15.186:0/1756075573 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660022020 con 0x7fa65c003500 2024-05-04T17:44:19.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:19.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:19.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 10 monclient: dump: 2024-05-04T17:44:19.639 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 10 monclient: _finish_auth 0 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 10 monclient: _check_auth_tickets 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 1 -- 172.21.15.186:0/1756075573 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa66000cc10 con 0x7fa65c003500 2024-05-04T17:44:19.640 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa675504640 5 monclient: authenticate success, global_id 4123 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa675504640 10 monclient: _renew_subs 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa675504640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa675504640 1 -- 172.21.15.186:0/1756075573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa670140c30 con 0x7fa65c003500 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 1 -- 172.21.15.186:0/1756075573 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660017040 con 0x7fa65c003500 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa6677fe640 10 monclient: dump: 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:19.641 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa66d7fa640 4 set_mon_vals no callback set 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa675504640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa675504640 10 monclient: _renew_subs 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa675504640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa675504640 1 -- 172.21.15.186:0/1756075573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa670138ed0 con 0x7fa65c003500 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.637+0000 7fa675504640 1 librados: init done 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.638+0000 7fa675504640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.638+0000 7fa675504640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.638+0000 7fa675504640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:19.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.638+0000 7fa675504640 1 -- 172.21.15.186:0/1756075573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa67006a700 con 0x7fa65c003500 2024-05-04T17:44:19.664 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:19.663+0000 7f33478fa040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-04T17:44:19.711 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:19.710+0000 7f33478fa040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-04T17:44:19.810 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:19.809+0000 7f9ef103e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-04T17:44:19.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.827+0000 7fa6677fe640 1 -- 172.21.15.186:0/1756075573 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa660025020 con 0x7fa65c003500 2024-05-04T17:44:19.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.827+0000 7fa6677fe640 1 -- 172.21.15.186:0/1756075573 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fa660031530 con 0x7fa65c003500 2024-05-04T17:44:19.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.827+0000 7fa6677fe640 10 client.4123.objecter ms_dispatch 0x7fa670000b90 osd_map(2..2 src has 1..2) 2024-05-04T17:44:19.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.827+0000 7fa6677fe640 3 client.4123.objecter handle_osd_map got epochs [2,2] > 0 2024-05-04T17:44:19.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.827+0000 7fa6677fe640 3 client.4123.objecter handle_osd_map decoding full epoch 2 2024-05-04T17:44:19.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.827+0000 7fa6677fe640 20 client.4123.objecter dump_active .. 0 homeless 2024-05-04T17:44:19.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.827+0000 7fa6677fe640 1 -- 172.21.15.186:0/1756075573 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa66002c020 con 0x7fa65c003500 2024-05-04T17:44:19.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.827+0000 7fa6677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:19.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.827+0000 7fa6677fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:19.873 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:19.872+0000 7f9ef103e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-04T17:44:19.883 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:19.883+0000 7f33478fa040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-04T17:44:19.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.901+0000 7fa675504640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4b749ec0-22b6-458c-a7af-2ccace806ab7", "id": 0}] 2024-05-04T17:44:19.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.901+0000 7fa675504640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4b749ec0-22b6-458c-a7af-2ccace806ab7", "id": 0}] 2024-05-04T17:44:19.902 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.901+0000 7fa675504640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:19.903 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:19.901+0000 7fa675504640 1 -- 172.21.15.186:0/1756075573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4b749ec0-22b6-458c-a7af-2ccace806ab7", "id": 0} v 0) -- 0x7fa67013aa70 con 0x7fa65c003500 2024-05-04T17:44:19.929 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:19.928+0000 7f9ef103e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-04T17:44:19.951 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:19.950+0000 7f33478fa040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-04T17:44:20.005 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:20.004+0000 7f33478fa040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-04T17:44:20.052 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:20.052+0000 7f9ef103e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-04T17:44:20.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.108+0000 7fa6677fe640 1 -- 172.21.15.186:0/1756075573 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4b749ec0-22b6-458c-a7af-2ccace806ab7", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fa6600036c0 con 0x7fa65c003500 2024-05-04T17:44:20.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.108+0000 7fa6677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4b749ec0-22b6-458c-a7af-2ccace806ab7", "id": 0}] 2024-05-04T17:44:20.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.108+0000 7fa6677fe640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:20.109 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-05-04T17:44:20.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 10 librados: watch_flush enter 2024-05-04T17:44:20.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 10 librados: watch_flush exit 2024-05-04T17:44:20.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 20 client.4123.objecter shutdown clearing up homeless session... 2024-05-04T17:44:20.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 10 client.4123.objecter successfully canceled tick 2024-05-04T17:44:20.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 10 monclient: shutdown 2024-05-04T17:44:20.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:20.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 1 -- 172.21.15.186:0/1756075573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa65c003500 msgr2=0x7fa6700b7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:20.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 1 --2- 172.21.15.186:0/1756075573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa65c003500 0x7fa6700b7540 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa660001390 tx=0x7fa660000fb0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 1 -- 172.21.15.186:0/1756075573 shutdown_connections 2024-05-04T17:44:20.111 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 1 --2- 172.21.15.186:0/1756075573 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa670138a50 0x7fa670132570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 1 --2- 172.21.15.186:0/1756075573 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa6701dbb80 0x7fa6701393d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 1 --2- 172.21.15.186:0/1756075573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa65c003500 0x7fa6700b7540 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 1 -- 172.21.15.186:0/1756075573 >> 172.21.15.186:0/1756075573 conn(0x7fa670067780 msgr2=0x7fa670074a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:20.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 1 -- 172.21.15.186:0/1756075573 shutdown_connections 2024-05-04T17:44:20.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 1 -- 172.21.15.186:0/1756075573 wait complete. 2024-05-04T17:44:20.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.109+0000 7fa675504640 1 librados: shutdown 2024-05-04T17:44:20.120 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:20.119+0000 7f33478fa040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-04T17:44:20.120 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 3aa520fe-39c4-4fa2-860b-f4d0cc04ca97 1 2024-05-04T17:44:20.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.216+0000 7f6718150640 10 monclient: get_monmap_and_config 2024-05-04T17:44:20.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.216+0000 7f6718150640 10 monclient: build_initial_monmap 2024-05-04T17:44:20.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.216+0000 7f6718150640 10 monclient: monmap: 2024-05-04T17:44:20.218 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:20.218 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:20.218 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:44:20.217556+0000 2024-05-04T17:44:20.218 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:44:20.217556+0000 2024-05-04T17:44:20.218 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:20.218 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:20.218 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:20.219 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:20.219 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:20.219 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:20.219 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:20.219 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:20.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.217+0000 7f6718150640 1 Processor -- start 2024-05-04T17:44:20.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.217+0000 7f6718150640 1 -- start start 2024-05-04T17:44:20.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.217+0000 7f6718150640 10 monclient: init 2024-05-04T17:44:20.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.217+0000 7f6718150640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:20.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.217+0000 7f6718150640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-05-04T17:44:20.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.217+0000 7f6718150640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f67100a9720 0x7f67100aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:20.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.217+0000 7f6718150640 10 monclient(hunting): picked mon.noname-b con 0x7f67100a9720 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:20.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.217+0000 7f6718150640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f67100a9b00 0x7f67100b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:20.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f6718150640 10 monclient(hunting): picked mon.noname-a con 0x7f67100a9b00 addr v2:172.21.15.19:3300/0 2024-05-04T17:44:20.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f6718150640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6710146890 0x7f671013fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:20.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f6718150640 10 monclient(hunting): picked mon.noname-c con 0x7f6710146890 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:20.223 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f6718150640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:20.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f6718150640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f67100abc40 con 0x7f67100a9b00 2024-05-04T17:44:20.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f6718150640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:20.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f6718150640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6710146090 con 0x7f6710146890 2024-05-04T17:44:20.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f6718150640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:20.224 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f6718150640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f6710146370 con 0x7f67100a9720 2024-05-04T17:44:20.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f6718150640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:20.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f6718150640 10 monclient(hunting): authenticate will time out at 1256.495617s 2024-05-04T17:44:20.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f6715eec640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f67100a9720 0x7f67100aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:20.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f67156eb640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f67100a9b00 0x7f67100b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:20.225 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.218+0000 7f67166ed640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6710146890 0x7f671013fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:20.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67156eb640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f67100a9b00 0x7f67100b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.186:44922/0 (socket says 172.21.15.186:44922) 2024-05-04T17:44:20.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67156eb640 1 -- 172.21.15.186:0/427072470 learned_addr learned my addr 172.21.15.186:0/427072470 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:20.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67156eb640 10 monclient(hunting): get_auth_request con 0x7f67100a9b00 auth_method 0 2024-05-04T17:44:20.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67156eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:20.226 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67156eb640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:20.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67156eb640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:20.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f6715eec640 10 monclient(hunting): get_auth_request con 0x7f67100a9720 auth_method 0 2024-05-04T17:44:20.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f6715eec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:20.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f6715eec640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:20.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f6715eec640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:20.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67166ed640 10 monclient(hunting): get_auth_request con 0x7f6710146890 auth_method 0 2024-05-04T17:44:20.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67166ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:20.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67166ed640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:20.227 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67166ed640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:20.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67156eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:20.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67156eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:20.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67156eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:20.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67166ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:20.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67166ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:20.228 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.219+0000 7f67166ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:20.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.220+0000 7f6715eec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:20.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.220+0000 7f6715eec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:20.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.220+0000 7f6715eec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:20.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.221+0000 7f67156eb640 10 monclient(hunting): handle_auth_done global_id 4113 payload 274 2024-05-04T17:44:20.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.221+0000 7f67156eb640 1 -- 172.21.15.186:0/427072470 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f67100a9720 msgr2=0x7f67100aa890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:20.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.221+0000 7f67156eb640 1 --2- 172.21.15.186:0/427072470 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f67100a9720 0x7f67100aa890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.221+0000 7f67156eb640 1 -- 172.21.15.186:0/427072470 >> v2:172.21.15.186:3300/0 conn(0x7f6710146890 msgr2=0x7f671013fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:20.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.221+0000 7f67156eb640 1 --2- 172.21.15.186:0/427072470 >> v2:172.21.15.186:3300/0 conn(0x7f6710146890 0x7f671013fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.221+0000 7f67156eb640 10 monclient: _finish_hunting 0 2024-05-04T17:44:20.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.221+0000 7f67156eb640 1 monclient: found mon.noname-a 2024-05-04T17:44:20.229 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.221+0000 7f67156eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:20.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.221+0000 7f67156eb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:44:20.230 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.221+0000 7f67156eb640 1 -- 172.21.15.186:0/427072470 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6710146d20 con 0x7f67100a9b00 2024-05-04T17:44:20.234 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.223+0000 7f67156eb640 1 --2- 172.21.15.186:0/427072470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67100a9b00 0x7f67100b2e90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f671006d760 tx=0x7f670002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7aabee7f727909 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:20.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 1 -- 172.21.15.186:0/427072470 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6700035020 con 0x7f67100a9b00 2024-05-04T17:44:20.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:20.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:44:20.235 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 10 monclient: dump: 2024-05-04T17:44:20.235 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:20.235 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:20.235 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.235 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.235 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:20.235 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:20.235 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:20.235 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 10 monclient: _finish_auth 0 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 10 monclient: _check_auth_tickets 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 1 -- 172.21.15.186:0/427072470 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6700031870 con 0x7f67100a9b00 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 1 -- 172.21.15.186:0/427072470 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6700031c40 con 0x7f67100a9b00 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6714eea640 10 monclient: dump: 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.236 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6718150640 5 monclient: authenticate success, global_id 4113 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6718150640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.227+0000 7f6718150640 4 set_mon_vals no callback set 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.228+0000 7f6718150640 10 monclient: shutdown 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.228+0000 7f6718150640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.228+0000 7f6718150640 1 -- 172.21.15.186:0/427072470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67100a9b00 msgr2=0x7f67100b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.228+0000 7f6718150640 1 --2- 172.21.15.186:0/427072470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67100a9b00 0x7f67100b2e90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f671006d760 tx=0x7f670002b910 comp rx=0 tx=0).stop 2024-05-04T17:44:20.237 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.228+0000 7f6718150640 1 -- 172.21.15.186:0/427072470 shutdown_connections 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.228+0000 7f6718150640 1 --2- 172.21.15.186:0/427072470 >> v2:172.21.15.186:3300/0 conn(0x7f6710146890 0x7f671013fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.228+0000 7f6718150640 1 --2- 172.21.15.186:0/427072470 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67100a9b00 0x7f67100b2e90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.228+0000 7f6718150640 1 --2- 172.21.15.186:0/427072470 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f67100a9720 0x7f67100aa890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.228+0000 7f6718150640 1 -- 172.21.15.186:0/427072470 >> 172.21.15.186:0/427072470 conn(0x7f6710067780 msgr2=0x7f67100b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.228+0000 7f6718150640 1 -- 172.21.15.186:0/427072470 shutdown_connections 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.228+0000 7f6718150640 1 -- 172.21.15.186:0/427072470 wait complete. 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.229+0000 7f6718150640 10 monclient: build_initial_monmap 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.229+0000 7f6718150640 10 monclient: monmap: 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:20.238 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.229+0000 7f6718150640 1 librados: starting msgr at 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.229+0000 7f6718150640 1 librados: starting objecter 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.229+0000 7f6718150640 1 Processor -- start 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.229+0000 7f6718150640 1 -- start start 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.229+0000 7f6718150640 1 librados: setting wanted keys 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.229+0000 7f6718150640 1 librados: calling monclient init 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.229+0000 7f6718150640 10 monclient: init 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f67100a9720 0x7f67100ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 10 monclient(hunting): picked mon.noname-c con 0x7f67100a9720 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:20.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67100a9b00 0x7f67101d7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 10 monclient(hunting): picked mon.noname-a con 0x7f67100a9b00 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6710146890 0x7f67101d84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 10 monclient(hunting): picked mon.noname-b con 0x7f6710146890 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6710140770 con 0x7f67100a9b00 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f67101405f0 con 0x7f6710146890 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f6710142b50 con 0x7f67100a9720 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f6718150640 10 monclient(hunting): authenticate will time out at 1256.507927s 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f67166ed640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6710146890 0x7f67101d84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:20.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.230+0000 7f67156eb640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67100a9b00 0x7f67101d7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f6715eec640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f67100a9720 0x7f67100ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f6715eec640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f67100a9720 0x7f67100ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.186:57488/0 (socket says 172.21.15.186:57488) 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f6715eec640 1 -- 172.21.15.186:0/3686164610 learned_addr learned my addr 172.21.15.186:0/3686164610 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f6715eec640 10 monclient(hunting): get_auth_request con 0x7f67100a9720 auth_method 0 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f6715eec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f6715eec640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f6715eec640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f67166ed640 10 monclient(hunting): get_auth_request con 0x7f6710146890 auth_method 0 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f67166ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f67166ed640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f67166ed640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f67156eb640 10 monclient(hunting): get_auth_request con 0x7f67100a9b00 auth_method 0 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f67156eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f67156eb640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:20.241 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f67156eb640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:20.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f6715eec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:20.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f6715eec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:20.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.231+0000 7f6715eec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:20.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.232+0000 7f67156eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:20.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.232+0000 7f67156eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:20.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.232+0000 7f67156eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:20.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.232+0000 7f67166ed640 1 -- 172.21.15.186:0/3686164610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6710146890 msgr2=0x7f67101d84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-04T17:44:20.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.232+0000 7f67166ed640 1 -- 172.21.15.186:0/3686164610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6710146890 msgr2=0x7f67101d84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-04T17:44:20.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.232+0000 7f67166ed640 1 --2- 172.21.15.186:0/3686164610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6710146890 0x7f67101d84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-04T17:44:20.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.232+0000 7f67166ed640 1 --2- 172.21.15.186:0/3686164610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6710146890 0x7f67101d84b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-04T17:44:20.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.234+0000 7f6715eec640 10 monclient(hunting): handle_auth_done global_id 4115 payload 824 2024-05-04T17:44:20.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.234+0000 7f6715eec640 1 -- 172.21.15.186:0/3686164610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6710146890 msgr2=0x7f67101d84b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-04T17:44:20.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.234+0000 7f6715eec640 1 --2- 172.21.15.186:0/3686164610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6710146890 0x7f67101d84b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.234+0000 7f6715eec640 1 -- 172.21.15.186:0/3686164610 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67100a9b00 msgr2=0x7f67101d7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:20.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.234+0000 7f6715eec640 1 --2- 172.21.15.186:0/3686164610 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67100a9b00 0x7f67101d7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.234+0000 7f6715eec640 10 monclient: _finish_hunting 0 2024-05-04T17:44:20.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.234+0000 7f6715eec640 1 monclient: found mon.noname-c 2024-05-04T17:44:20.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.234+0000 7f6715eec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:20.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.234+0000 7f6715eec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.234+0000 7f6715eec640 1 -- 172.21.15.186:0/3686164610 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67100ada40 con 0x7f67100a9720 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.237+0000 7f6715eec640 1 --2- 172.21.15.186:0/3686164610 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f67100a9720 0x7f67100ac870 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f670c00e890 tx=0x7f670c00ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.237+0000 7f67067fc640 10 client.?.objecter ms_handle_connect 0x7f67100a9720 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.237+0000 7f67067fc640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 1 -- 172.21.15.186:0/3686164610 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c017070 con 0x7f67100a9720 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 10 monclient: dump: 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:20.245 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 10 monclient: _finish_auth 0 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 10 monclient: _check_auth_tickets 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 1 -- 172.21.15.186:0/3686164610 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f670c00c940 con 0x7f67100a9720 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 1 -- 172.21.15.186:0/3686164610 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c00cd10 con 0x7f67100a9720 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67067fc640 10 monclient: dump: 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:20.246 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f6718150640 5 monclient: authenticate success, global_id 4115 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f6718150640 10 monclient: _renew_subs 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f6718150640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f6718150640 1 -- 172.21.15.186:0/3686164610 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67100aec60 con 0x7f67100a9720 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f67077fe640 4 set_mon_vals no callback set 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f6718150640 10 client.4115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f6718150640 10 monclient: _renew_subs 2024-05-04T17:44:20.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f6718150640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:44:20.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f6718150640 1 -- 172.21.15.186:0/3686164610 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f67101d89f0 con 0x7f67100a9720 2024-05-04T17:44:20.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.240+0000 7f6718150640 1 librados: init done 2024-05-04T17:44:20.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.241+0000 7f6718150640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:20.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.241+0000 7f6718150640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:20.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.241+0000 7f6718150640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:44:20.248 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.241+0000 7f6718150640 1 -- 172.21.15.186:0/3686164610 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f671006a700 con 0x7f67100a9720 2024-05-04T17:44:20.348 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.347+0000 7f67067fc640 1 -- 172.21.15.186:0/3686164610 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f670c01a070 con 0x7f67100a9720 2024-05-04T17:44:20.352 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.351+0000 7f67067fc640 1 -- 172.21.15.186:0/3686164610 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f670c013470 con 0x7f67100a9720 2024-05-04T17:44:20.353 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.351+0000 7f67067fc640 10 client.4115.objecter ms_dispatch 0x7f6710000b90 osd_map(3..3 src has 1..3) 2024-05-04T17:44:20.353 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.351+0000 7f67067fc640 3 client.4115.objecter handle_osd_map got epochs [3,3] > 0 2024-05-04T17:44:20.353 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.351+0000 7f67067fc640 3 client.4115.objecter handle_osd_map decoding full epoch 3 2024-05-04T17:44:20.353 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.351+0000 7f67067fc640 20 client.4115.objecter dump_active .. 0 homeless 2024-05-04T17:44:20.353 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.352+0000 7f67067fc640 1 -- 172.21.15.186:0/3686164610 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f670c005990 con 0x7f67100a9720 2024-05-04T17:44:20.353 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.352+0000 7f67067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:20.353 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.352+0000 7f67067fc640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:20.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.424+0000 7f6718150640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3aa520fe-39c4-4fa2-860b-f4d0cc04ca97", "id": 1}] 2024-05-04T17:44:20.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.424+0000 7f6718150640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3aa520fe-39c4-4fa2-860b-f4d0cc04ca97", "id": 1}] 2024-05-04T17:44:20.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.424+0000 7f6718150640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:44:20.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.424+0000 7f6718150640 1 -- 172.21.15.186:0/3686164610 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3aa520fe-39c4-4fa2-860b-f4d0cc04ca97", "id": 1} v 0) -- 0x7f67100acdb0 con 0x7f67100a9720 2024-05-04T17:44:20.466 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:20.466+0000 7f9ef103e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-04T17:44:20.514 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:20.513+0000 7f33478fa040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-04T17:44:20.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.517+0000 7f67067fc640 1 -- 172.21.15.186:0/3686164610 <== mon.2 v2:172.21.15.19:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3aa520fe-39c4-4fa2-860b-f4d0cc04ca97", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f670c005c40 con 0x7f67100a9720 2024-05-04T17:44:20.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.517+0000 7f67067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3aa520fe-39c4-4fa2-860b-f4d0cc04ca97", "id": 1}] 2024-05-04T17:44:20.518 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.517+0000 7f67067fc640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:20.518 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-05-04T17:44:20.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 10 librados: watch_flush enter 2024-05-04T17:44:20.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 10 librados: watch_flush exit 2024-05-04T17:44:20.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 20 client.4115.objecter shutdown clearing up homeless session... 2024-05-04T17:44:20.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 10 client.4115.objecter successfully canceled tick 2024-05-04T17:44:20.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 10 monclient: shutdown 2024-05-04T17:44:20.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:20.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 1 -- 172.21.15.186:0/3686164610 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f67100a9720 msgr2=0x7f67100ac870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:20.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 1 --2- 172.21.15.186:0/3686164610 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f67100a9720 0x7f67100ac870 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f670c00e890 tx=0x7f670c00ed50 comp rx=0 tx=0).stop 2024-05-04T17:44:20.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 1 -- 172.21.15.186:0/3686164610 shutdown_connections 2024-05-04T17:44:20.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 1 --2- 172.21.15.186:0/3686164610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6710146890 0x7f67101d84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 1 --2- 172.21.15.186:0/3686164610 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f67100a9b00 0x7f67101d7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 1 --2- 172.21.15.186:0/3686164610 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f67100a9720 0x7f67100ac870 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 1 -- 172.21.15.186:0/3686164610 >> 172.21.15.186:0/3686164610 conn(0x7f6710067780 msgr2=0x7f671013ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:20.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 1 -- 172.21.15.186:0/3686164610 shutdown_connections 2024-05-04T17:44:20.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 1 -- 172.21.15.186:0/3686164610 wait complete. 2024-05-04T17:44:20.521 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.518+0000 7f6718150640 1 librados: shutdown 2024-05-04T17:44:20.529 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 16ed21c9-e5c3-4d65-b4d1-e4927a0fe919 2 2024-05-04T17:44:20.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.625+0000 7f2a34a6d640 10 monclient: get_monmap_and_config 2024-05-04T17:44:20.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.625+0000 7f2a34a6d640 10 monclient: build_initial_monmap 2024-05-04T17:44:20.626 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.625+0000 7f2a34a6d640 10 monclient: monmap: 2024-05-04T17:44:20.626 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:20.626 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:20.626 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:44:20.625948+0000 2024-05-04T17:44:20.626 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:44:20.625948+0000 2024-05-04T17:44:20.626 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:20.626 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:20.626 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:20.627 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:20.627 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:20.627 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:20.627 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:20.627 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:20.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.625+0000 7f2a34a6d640 1 Processor -- start 2024-05-04T17:44:20.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.625+0000 7f2a34a6d640 1 -- start start 2024-05-04T17:44:20.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.625+0000 7f2a34a6d640 10 monclient: init 2024-05-04T17:44:20.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:20.629 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f2a300a9720 0x7f2a300aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 10 monclient(hunting): picked mon.noname-a con 0x7f2a300a9720 addr v2:172.21.15.19:3300/0 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a300a9b00 addr v1:172.21.15.19:6789/0 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2a300b2e90 0x7f2a3013fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 10 monclient(hunting): picked mon.noname-b con 0x7f2a300b2e90 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a300abc40 con 0x7f2a300a9b00 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f2a30146fa0 con 0x7f2a300a9720 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f2a30146090 con 0x7f2a300b2e90 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.626+0000 7f2a34a6d640 10 monclient(hunting): authenticate will time out at 1256.904010s 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.627+0000 7f2a2ed9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2a300b2e90 0x7f2a3013fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.627+0000 7f2a2e59c640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f2a300a9720 0x7f2a300aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:20.630 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.627+0000 7f2a2ed9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2a300b2e90 0x7f2a3013fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.186:57498/0 (socket says 172.21.15.186:57498) 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.627+0000 7f2a27fff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f2a300a9b00 0x7f2a30146890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.186:49166/0 (socket says 172.21.15.186:49166) 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.627+0000 7f2a2ed9d640 1 -- 172.21.15.186:0/2693188478 learned_addr learned my addr 172.21.15.186:0/2693188478 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.627+0000 7f2a2ed9d640 10 monclient(hunting): get_auth_request con 0x7f2a300b2e90 auth_method 0 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.627+0000 7f2a2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.627+0000 7f2a2ed9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.627+0000 7f2a2ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.627+0000 7f2a2e59c640 10 monclient(hunting): get_auth_request con 0x7f2a300a9720 auth_method 0 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.627+0000 7f2a2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.627+0000 7f2a2e59c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.628+0000 7f2a2e59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.628+0000 7f2a2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.628+0000 7f2a2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:20.631 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.628+0000 7f2a2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:20.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.628+0000 7f2a2e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:20.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.628+0000 7f2a2e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:20.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.628+0000 7f2a2e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:20.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.630+0000 7f2a2ed9d640 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2024-05-04T17:44:20.632 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.630+0000 7f2a2ed9d640 1 -- 172.21.15.186:0/2693188478 >> v2:172.21.15.19:3300/0 conn(0x7f2a300a9720 msgr2=0x7f2a300aa890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:20.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.630+0000 7f2a2ed9d640 1 --2- 172.21.15.186:0/2693188478 >> v2:172.21.15.19:3300/0 conn(0x7f2a300a9720 0x7f2a300aa890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.630+0000 7f2a2ed9d640 1 -- 172.21.15.186:0/2693188478 >> v1:172.21.15.19:6789/0 conn(0x7f2a300a9b00 legacy=0x7f2a30146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:20.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.630+0000 7f2a2ed9d640 10 monclient: _finish_hunting 0 2024-05-04T17:44:20.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.630+0000 7f2a2ed9d640 1 monclient: found mon.noname-b 2024-05-04T17:44:20.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.630+0000 7f2a2ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:20.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.630+0000 7f2a2ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:44:20.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.630+0000 7f2a2ed9d640 1 -- 172.21.15.186:0/2693188478 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a30146370 con 0x7f2a300b2e90 2024-05-04T17:44:20.633 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.632+0000 7f2a2ed9d640 1 --2- 172.21.15.186:0/2693188478 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2a300b2e90 0x7f2a3013fea0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009e50 tx=0x7f2a2802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=703227b498d29301 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:20.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.634+0000 7f2a2dd9b640 1 -- 172.21.15.186:0/2693188478 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2802cb00 con 0x7f2a300b2e90 2024-05-04T17:44:20.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.634+0000 7f2a2dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:20.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.634+0000 7f2a2dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:20.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.634+0000 7f2a2dd9b640 10 monclient: dump: 2024-05-04T17:44:20.635 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:20.635 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:20.635 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.636 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.636 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:20.636 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:20.636 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:20.636 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:20.636 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:20.636 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:20.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.634+0000 7f2a2dd9b640 10 monclient: _finish_auth 0 2024-05-04T17:44:20.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.634+0000 7f2a2dd9b640 10 monclient: _check_auth_tickets 2024-05-04T17:44:20.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.634+0000 7f2a2dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:20.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.634+0000 7f2a34a6d640 5 monclient: authenticate success, global_id 4118 2024-05-04T17:44:20.636 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.634+0000 7f2a34a6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:44:20.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.635+0000 7f2a2dd9b640 1 -- 172.21.15.186:0/2693188478 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a2802cca0 con 0x7f2a300b2e90 2024-05-04T17:44:20.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.635+0000 7f2a2dd9b640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:20.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.635+0000 7f2a2dd9b640 1 -- 172.21.15.186:0/2693188478 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28034610 con 0x7f2a300b2e90 2024-05-04T17:44:20.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.635+0000 7f2a2dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:20.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.635+0000 7f2a2dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:44:20.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.635+0000 7f2a2dd9b640 10 monclient: dump: 2024-05-04T17:44:20.638 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:20.638 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:20.638 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.638 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.638 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:20.638 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:20.638 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.635+0000 7f2a34a6d640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.635+0000 7f2a34a6d640 4 set_mon_vals no callback set 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.636+0000 7f2a34a6d640 10 monclient: shutdown 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.636+0000 7f2a34a6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.636+0000 7f2a34a6d640 1 -- 172.21.15.186:0/2693188478 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2a300b2e90 msgr2=0x7f2a3013fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.636+0000 7f2a34a6d640 1 --2- 172.21.15.186:0/2693188478 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2a300b2e90 0x7f2a3013fea0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009e50 tx=0x7f2a2802bf60 comp rx=0 tx=0).stop 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.636+0000 7f2a34a6d640 1 -- 172.21.15.186:0/2693188478 shutdown_connections 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.636+0000 7f2a34a6d640 1 --2- 172.21.15.186:0/2693188478 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2a300b2e90 0x7f2a3013fea0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.636+0000 7f2a34a6d640 1 --2- 172.21.15.186:0/2693188478 >> v2:172.21.15.19:3300/0 conn(0x7f2a300a9720 0x7f2a300aa890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.636+0000 7f2a34a6d640 1 -- 172.21.15.186:0/2693188478 >> 172.21.15.186:0/2693188478 conn(0x7f2a30067780 msgr2=0x7f2a300b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.636+0000 7f2a34a6d640 1 -- 172.21.15.186:0/2693188478 shutdown_connections 2024-05-04T17:44:20.639 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.636+0000 7f2a34a6d640 1 -- 172.21.15.186:0/2693188478 wait complete. 2024-05-04T17:44:20.642 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.637+0000 7f2a34a6d640 10 monclient: build_initial_monmap 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.637+0000 7f2a34a6d640 10 monclient: monmap: 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.637+0000 7f2a34a6d640 1 librados: starting msgr at 2024-05-04T17:44:20.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.637+0000 7f2a34a6d640 1 librados: starting objecter 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.637+0000 7f2a34a6d640 1 Processor -- start 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.637+0000 7f2a34a6d640 1 -- start start 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.637+0000 7f2a34a6d640 1 librados: setting wanted keys 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.637+0000 7f2a34a6d640 1 librados: calling monclient init 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.637+0000 7f2a34a6d640 10 monclient: init 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a300a9720 0x7f2a301db840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 10 monclient(hunting): picked mon.noname-a con 0x7f2a300a9720 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2a300a9b00 0x7f2a30139650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 10 monclient(hunting): picked mon.noname-c con 0x7f2a300a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a300b2e90 0x7f2a30139b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 10 monclient(hunting): picked mon.noname-b con 0x7f2a300b2e90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:20.644 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2a301407b0 con 0x7f2a300a9720 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2a30140630 con 0x7f2a300b2e90 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f2a3014d420 con 0x7f2a300a9b00 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.638+0000 7f2a34a6d640 10 monclient(hunting): authenticate will time out at 1256.916026s 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a2e59c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a300a9720 0x7f2a301db840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a2ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a300b2e90 0x7f2a30139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a2e59c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a300a9720 0x7f2a301db840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.186:44950/0 (socket says 172.21.15.186:44950) 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a2e59c640 1 -- 172.21.15.186:0/680884016 learned_addr learned my addr 172.21.15.186:0/680884016 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a2e59c640 10 monclient(hunting): get_auth_request con 0x7f2a300a9720 auth_method 0 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a2e59c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a2e59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:20.645 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a27fff640 1 --2- 172.21.15.186:0/680884016 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2a300a9b00 0x7f2a30139650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a2ed9d640 10 monclient(hunting): get_auth_request con 0x7f2a300b2e90 auth_method 0 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a2ed9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a2ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a27fff640 10 monclient(hunting): get_auth_request con 0x7f2a300a9b00 auth_method 0 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a27fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.639+0000 7f2a27fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.640+0000 7f2a2e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.640+0000 7f2a2e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.640+0000 7f2a2e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.640+0000 7f2a2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.640+0000 7f2a2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.640+0000 7f2a2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:20.646 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.640+0000 7f2a27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:20.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.640+0000 7f2a27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:20.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.640+0000 7f2a27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:20.647 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.643+0000 7f2a2ed9d640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-05-04T17:44:20.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.643+0000 7f2a2ed9d640 1 -- 172.21.15.186:0/680884016 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2a300a9b00 msgr2=0x7f2a30139650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:20.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.643+0000 7f2a2ed9d640 1 --2- 172.21.15.186:0/680884016 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2a300a9b00 0x7f2a30139650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.643+0000 7f2a2ed9d640 1 -- 172.21.15.186:0/680884016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a300a9720 msgr2=0x7f2a301db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:20.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.643+0000 7f2a2ed9d640 1 --2- 172.21.15.186:0/680884016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a300a9720 0x7f2a301db840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.643+0000 7f2a2ed9d640 10 monclient: _finish_hunting 0 2024-05-04T17:44:20.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.643+0000 7f2a2ed9d640 1 monclient: found mon.noname-b 2024-05-04T17:44:20.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.643+0000 7f2a2ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:20.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.643+0000 7f2a2ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:20.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.643+0000 7f2a2ed9d640 1 -- 172.21.15.186:0/680884016 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a30149750 con 0x7f2a300b2e90 2024-05-04T17:44:20.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.645+0000 7f2a2ed9d640 1 --2- 172.21.15.186:0/680884016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a300b2e90 0x7f2a30139b90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2a28032000 tx=0x7f2a28004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:20.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.645+0000 7f2a26ffd640 10 client.?.objecter ms_handle_connect 0x7f2a300b2e90 2024-05-04T17:44:20.648 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.645+0000 7f2a26ffd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:20.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 1 -- 172.21.15.186:0/680884016 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28044020 con 0x7f2a300b2e90 2024-05-04T17:44:20.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:20.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:20.650 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 10 monclient: dump: 2024-05-04T17:44:20.650 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:20.650 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:20.650 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.650 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.650 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:20.650 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:20.650 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:20.650 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 10 monclient: _finish_auth 0 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 10 monclient: _check_auth_tickets 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 1 -- 172.21.15.186:0/680884016 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a2802ce50 con 0x7f2a300b2e90 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 1 -- 172.21.15.186:0/680884016 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28009510 con 0x7f2a300b2e90 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a26ffd640 10 monclient: dump: 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.651 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a2d59a640 4 set_mon_vals no callback set 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a34a6d640 5 monclient: authenticate success, global_id 4132 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a34a6d640 10 monclient: _renew_subs 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a34a6d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.648+0000 7f2a34a6d640 1 -- 172.21.15.186:0/680884016 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a30143890 con 0x7f2a300b2e90 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.649+0000 7f2a34a6d640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.649+0000 7f2a34a6d640 10 monclient: _renew_subs 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.649+0000 7f2a34a6d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:20.652 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.649+0000 7f2a34a6d640 1 -- 172.21.15.186:0/680884016 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a301dbf60 con 0x7f2a300b2e90 2024-05-04T17:44:20.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.649+0000 7f2a34a6d640 1 librados: init done 2024-05-04T17:44:20.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.650+0000 7f2a34a6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:20.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.650+0000 7f2a34a6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:20.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.650+0000 7f2a34a6d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:20.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.650+0000 7f2a34a6d640 1 -- 172.21.15.186:0/680884016 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a3006a700 con 0x7f2a300b2e90 2024-05-04T17:44:20.752 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.751+0000 7f2a26ffd640 1 -- 172.21.15.186:0/680884016 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2a28047020 con 0x7f2a300b2e90 2024-05-04T17:44:20.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.751+0000 7f2a26ffd640 1 -- 172.21.15.186:0/680884016 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f2a28052c00 con 0x7f2a300b2e90 2024-05-04T17:44:20.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.751+0000 7f2a26ffd640 10 client.4132.objecter ms_dispatch 0x7f2a30000b90 osd_map(4..4 src has 1..4) 2024-05-04T17:44:20.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.751+0000 7f2a26ffd640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-05-04T17:44:20.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.751+0000 7f2a26ffd640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-05-04T17:44:20.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.751+0000 7f2a26ffd640 20 client.4132.objecter dump_active .. 0 homeless 2024-05-04T17:44:20.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.751+0000 7f2a26ffd640 1 -- 172.21.15.186:0/680884016 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2a2804e030 con 0x7f2a300b2e90 2024-05-04T17:44:20.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.751+0000 7f2a26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:20.753 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.751+0000 7f2a26ffd640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:20.763 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:20.762+0000 7f9ef103e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-04T17:44:20.786 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:20.785+0000 7f33478fa040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-04T17:44:20.818 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:20.817+0000 7f9ef103e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-04T17:44:20.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.823+0000 7f2a34a6d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "16ed21c9-e5c3-4d65-b4d1-e4927a0fe919", "id": 2}] 2024-05-04T17:44:20.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.823+0000 7f2a34a6d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "16ed21c9-e5c3-4d65-b4d1-e4927a0fe919", "id": 2}] 2024-05-04T17:44:20.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.823+0000 7f2a34a6d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:20.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.823+0000 7f2a34a6d640 1 -- 172.21.15.186:0/680884016 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "16ed21c9-e5c3-4d65-b4d1-e4927a0fe919", "id": 2} v 0) -- 0x7f2a30143a60 con 0x7f2a300b2e90 2024-05-04T17:44:20.840 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:20.838+0000 7f33478fa040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-04T17:44:20.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.906+0000 7f2a26ffd640 1 -- 172.21.15.186:0/680884016 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "16ed21c9-e5c3-4d65-b4d1-e4927a0fe919", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f2a2802cb10 con 0x7f2a300b2e90 2024-05-04T17:44:20.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.906+0000 7f2a26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "16ed21c9-e5c3-4d65-b4d1-e4927a0fe919", "id": 2}] 2024-05-04T17:44:20.907 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.906+0000 7f2a26ffd640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:20.908 INFO:teuthology.orchestra.run.smithi186.stdout:2 2024-05-04T17:44:20.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 10 librados: watch_flush enter 2024-05-04T17:44:20.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 10 librados: watch_flush exit 2024-05-04T17:44:20.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 20 client.4132.objecter shutdown clearing up homeless session... 2024-05-04T17:44:20.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 10 client.4132.objecter successfully canceled tick 2024-05-04T17:44:20.909 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 10 monclient: shutdown 2024-05-04T17:44:20.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:20.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 1 -- 172.21.15.186:0/680884016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a300b2e90 msgr2=0x7f2a30139b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:20.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 1 --2- 172.21.15.186:0/680884016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a300b2e90 0x7f2a30139b90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2a28032000 tx=0x7f2a28004a40 comp rx=0 tx=0).stop 2024-05-04T17:44:20.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 1 -- 172.21.15.186:0/680884016 shutdown_connections 2024-05-04T17:44:20.910 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 1 --2- 172.21.15.186:0/680884016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a300b2e90 0x7f2a30139b90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 1 --2- 172.21.15.186:0/680884016 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2a300a9b00 0x7f2a30139650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 1 --2- 172.21.15.186:0/680884016 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2a300a9720 0x7f2a301db840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:20.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 1 -- 172.21.15.186:0/680884016 >> 172.21.15.186:0/680884016 conn(0x7f2a30067780 msgr2=0x7f2a300b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:20.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 1 -- 172.21.15.186:0/680884016 shutdown_connections 2024-05-04T17:44:20.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 1 -- 172.21.15.186:0/680884016 wait complete. 2024-05-04T17:44:20.911 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:20.907+0000 7f2a34a6d640 1 librados: shutdown 2024-05-04T17:44:20.919 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 45aa2c0f-cfae-478c-a60c-4c1491e45525 3 2024-05-04T17:44:20.963 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:20.963+0000 7f33478fa040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-04T17:44:20.964 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:20.963+0000 7f9ef103e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-04T17:44:21.015 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.014+0000 7f2787159640 10 monclient: get_monmap_and_config 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.014+0000 7f2787159640 10 monclient: build_initial_monmap 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.014+0000 7f2787159640 10 monclient: monmap: 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:44:21.015730+0000 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:44:21.015730+0000 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:21.016 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:21.017 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.015+0000 7f2787159640 1 Processor -- start 2024-05-04T17:44:21.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.015+0000 7f2787159640 1 -- start start 2024-05-04T17:44:21.018 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.015+0000 7f2787159640 10 monclient: init 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2787159640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2787159640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2787159640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f27800a9720 0x7f27800aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2787159640 10 monclient(hunting): picked mon.noname-c con 0x7f27800a9720 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2787159640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27800a9b00 addr v1:172.21.15.19:6789/0 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2787159640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27800b2e90 addr v1:172.21.15.186:6789/0 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2787159640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27800abc40 con 0x7f27800a9b00 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2787159640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2780146fa0 con 0x7f27800b2e90 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2787159640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2787159640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f2780146090 con 0x7f27800a9720 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2787159640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2787159640 10 monclient(hunting): authenticate will time out at 1257.293878s 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.016+0000 7f2784ef5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f27800a9720 0x7f27800aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.017+0000 7f277ffff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f27800a9b00 0x7f2780146890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.186:49180/0 (socket says 172.21.15.186:49180) 2024-05-04T17:44:21.019 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.017+0000 7f277ffff640 1 -- 172.21.15.186:0/1324624820 learned_addr learned my addr 172.21.15.186:0/1324624820 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-05-04T17:44:21.020 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.017+0000 7f2784ef5640 10 monclient(hunting): get_auth_request con 0x7f27800a9720 auth_method 0 2024-05-04T17:44:21.020 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.017+0000 7f2784ef5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.020 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.017+0000 7f2784ef5640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.020 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.017+0000 7f2784ef5640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.020 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.017+0000 7f2784ef5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.020 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.017+0000 7f2784ef5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.020 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.017+0000 7f2784ef5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.020 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.019+0000 7f2784ef5640 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2024-05-04T17:44:21.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.019+0000 7f2784ef5640 1 -- 172.21.15.186:0/1324624820 >> v1:172.21.15.186:6789/0 conn(0x7f27800b2e90 legacy=0x7f278013fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:21.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.019+0000 7f2784ef5640 1 -- 172.21.15.186:0/1324624820 >> v1:172.21.15.19:6789/0 conn(0x7f27800a9b00 legacy=0x7f2780146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.019+0000 7f2784ef5640 10 monclient: _finish_hunting 0 2024-05-04T17:44:21.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.019+0000 7f2784ef5640 1 monclient: found mon.noname-c 2024-05-04T17:44:21.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.019+0000 7f2784ef5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:21.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.019+0000 7f2784ef5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.021 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.019+0000 7f2784ef5640 1 -- 172.21.15.186:0/1324624820 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2780146370 con 0x7f27800a9720 2024-05-04T17:44:21.022 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.021+0000 7f2784ef5640 1 --2- 172.21.15.186:0/1324624820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27800a9720 0x7f27800aa890 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f27740097d0 tx=0x7f277402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcebae483652496f server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:21.026 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 1 -- 172.21.15.186:0/1324624820 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2774035020 con 0x7f27800a9720 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 10 monclient: dump: 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:21.027 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 10 monclient: _finish_auth 0 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 10 monclient: _check_auth_tickets 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 1 -- 172.21.15.186:0/1324624820 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2774031830 con 0x7f27800a9720 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 1 -- 172.21.15.186:0/1324624820 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2774031c00 con 0x7f27800a9720 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f2787159640 5 monclient: authenticate success, global_id 4135 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f2787159640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f2787159640 4 set_mon_vals no callback set 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.025+0000 7f277f7fe640 10 monclient: dump: 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:21.028 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.029 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.029 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.029 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:21.029 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.029 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:21.029 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:21.029 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:21.029 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.032 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.026+0000 7f2787159640 10 monclient: shutdown 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.026+0000 7f2787159640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.026+0000 7f2787159640 1 -- 172.21.15.186:0/1324624820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27800a9720 msgr2=0x7f27800aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.026+0000 7f2787159640 1 --2- 172.21.15.186:0/1324624820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27800a9720 0x7f27800aa890 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f27740097d0 tx=0x7f277402b8d0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.026+0000 7f2787159640 1 -- 172.21.15.186:0/1324624820 shutdown_connections 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.026+0000 7f2787159640 1 --2- 172.21.15.186:0/1324624820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27800a9720 0x7f27800aa890 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.026+0000 7f2787159640 1 -- 172.21.15.186:0/1324624820 >> 172.21.15.186:0/1324624820 conn(0x7f2780067780 msgr2=0x7f27800b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.026+0000 7f2787159640 1 -- 172.21.15.186:0/1324624820 shutdown_connections 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.026+0000 7f2787159640 1 -- 172.21.15.186:0/1324624820 wait complete. 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.027+0000 7f2787159640 10 monclient: build_initial_monmap 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.027+0000 7f2787159640 10 monclient: monmap: 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-05-04T17:44:21.033 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.027+0000 7f2787159640 1 librados: starting msgr at 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.027+0000 7f2787159640 1 librados: starting objecter 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.027+0000 7f2787159640 1 Processor -- start 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.027+0000 7f2787159640 1 -- start start 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.027+0000 7f2787159640 1 librados: setting wanted keys 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.027+0000 7f2787159640 1 librados: calling monclient init 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.027+0000 7f2787159640 10 monclient: init 2024-05-04T17:44:21.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f27800a9720 0x7f278013e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 10 monclient(hunting): picked mon.noname-c con 0x7f27800a9720 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27800a9b00 0x7f2780143e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 10 monclient(hunting): picked mon.noname-a con 0x7f27800a9b00 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27800b2e90 0x7f27801443d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 10 monclient(hunting): picked mon.noname-b con 0x7f27800b2e90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f278014e430 con 0x7f27800a9b00 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f278014e5b0 con 0x7f27800b2e90 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f278014e730 con 0x7f27800a9720 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:21.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.028+0000 7f2787159640 10 monclient(hunting): authenticate will time out at 1257.306305s 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f27856f6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27800b2e90 0x7f27801443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f277ffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27800a9b00 0x7f2780143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f2784ef5640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f27800a9720 0x7f278013e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f27856f6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27800b2e90 0x7f27801443d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52744/0 (socket says 172.21.15.186:52744) 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f27856f6640 1 -- 172.21.15.186:0/3418749626 learned_addr learned my addr 172.21.15.186:0/3418749626 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f27856f6640 10 monclient(hunting): get_auth_request con 0x7f27800b2e90 auth_method 0 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f27856f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f27856f6640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f27856f6640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f277ffff640 10 monclient(hunting): get_auth_request con 0x7f27800a9b00 auth_method 0 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f277ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f277ffff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.029+0000 7f277ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f27856f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f27856f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.036 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f27856f6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f2784ef5640 10 monclient(hunting): get_auth_request con 0x7f27800a9720 auth_method 0 2024-05-04T17:44:21.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f2784ef5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f2784ef5640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f2784ef5640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f277ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f277ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f277ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f2784ef5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f2784ef5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.037 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.030+0000 7f2784ef5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.032+0000 7f27856f6640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-05-04T17:44:21.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.032+0000 7f27856f6640 1 -- 172.21.15.186:0/3418749626 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f27800a9720 msgr2=0x7f278013e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.032+0000 7f27856f6640 1 --2- 172.21.15.186:0/3418749626 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f27800a9720 0x7f278013e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.032+0000 7f27856f6640 1 -- 172.21.15.186:0/3418749626 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27800a9b00 msgr2=0x7f2780143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.032+0000 7f27856f6640 1 --2- 172.21.15.186:0/3418749626 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27800a9b00 0x7f2780143e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.032+0000 7f27856f6640 10 monclient: _finish_hunting 0 2024-05-04T17:44:21.038 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.032+0000 7f27856f6640 1 monclient: found mon.noname-b 2024-05-04T17:44:21.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.032+0000 7f27856f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:21.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.032+0000 7f27856f6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.032+0000 7f27856f6640 1 -- 172.21.15.186:0/3418749626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2780142190 con 0x7f27800b2e90 2024-05-04T17:44:21.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.035+0000 7f27856f6640 1 --2- 172.21.15.186:0/3418749626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27800b2e90 0x7f27801443d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f277800b990 tx=0x7f277800be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:21.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.035+0000 7f277d7fa640 10 client.?.objecter ms_handle_connect 0x7f27800b2e90 2024-05-04T17:44:21.039 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.035+0000 7f277d7fa640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:21.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 1 -- 172.21.15.186:0/3418749626 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778019070 con 0x7f27800b2e90 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 10 monclient: dump: 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 10 monclient: _finish_auth 0 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 10 monclient: _check_auth_tickets 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 1 -- 172.21.15.186:0/3418749626 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f277800cba0 con 0x7f27800b2e90 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f2787159640 5 monclient: authenticate success, global_id 4138 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f2787159640 10 monclient: _renew_subs 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 1 -- 172.21.15.186:0/3418749626 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778014440 con 0x7f27800b2e90 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f2787159640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f2787159640 1 -- 172.21.15.186:0/3418749626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2780132200 con 0x7f27800b2e90 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277effd640 4 set_mon_vals no callback set 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f2787159640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:21.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f277d7fa640 10 monclient: dump: 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f2787159640 10 monclient: _renew_subs 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f2787159640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f2787159640 1 -- 172.21.15.186:0/3418749626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2780140f10 con 0x7f27800b2e90 2024-05-04T17:44:21.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.038+0000 7f2787159640 1 librados: init done 2024-05-04T17:44:21.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.039+0000 7f2787159640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:21.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.039+0000 7f2787159640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:21.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.039+0000 7f2787159640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.039+0000 7f2787159640 1 -- 172.21.15.186:0/3418749626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f278006a700 con 0x7f27800b2e90 2024-05-04T17:44:21.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.143+0000 7f277d7fa640 1 -- 172.21.15.186:0/3418749626 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f277801c070 con 0x7f27800b2e90 2024-05-04T17:44:21.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.143+0000 7f277d7fa640 1 -- 172.21.15.186:0/3418749626 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f277801d840 con 0x7f27800b2e90 2024-05-04T17:44:21.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.143+0000 7f277d7fa640 10 client.4138.objecter ms_dispatch 0x7f2780000b90 osd_map(5..5 src has 1..5) 2024-05-04T17:44:21.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.143+0000 7f277d7fa640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-05-04T17:44:21.145 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.143+0000 7f277d7fa640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-05-04T17:44:21.145 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.143+0000 7f277d7fa640 20 client.4138.objecter dump_active .. 0 homeless 2024-05-04T17:44:21.145 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.143+0000 7f277d7fa640 1 -- 172.21.15.186:0/3418749626 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2778023020 con 0x7f27800b2e90 2024-05-04T17:44:21.145 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.143+0000 7f277d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:21.145 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.143+0000 7f277d7fa640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:21.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.219+0000 7f2787159640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "45aa2c0f-cfae-478c-a60c-4c1491e45525", "id": 3}] 2024-05-04T17:44:21.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.219+0000 7f2787159640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "45aa2c0f-cfae-478c-a60c-4c1491e45525", "id": 3}] 2024-05-04T17:44:21.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.219+0000 7f2787159640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.219+0000 7f2787159640 1 -- 172.21.15.186:0/3418749626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "45aa2c0f-cfae-478c-a60c-4c1491e45525", "id": 3} v 0) -- 0x7f27801407c0 con 0x7f27800b2e90 2024-05-04T17:44:21.253 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:21.252+0000 7f33478fa040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-04T17:44:21.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.279+0000 7f277d7fa640 1 -- 172.21.15.186:0/3418749626 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "45aa2c0f-cfae-478c-a60c-4c1491e45525", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f277801d560 con 0x7f27800b2e90 2024-05-04T17:44:21.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.279+0000 7f277d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "45aa2c0f-cfae-478c-a60c-4c1491e45525", "id": 3}] 2024-05-04T17:44:21.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.279+0000 7f277d7fa640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:21.281 INFO:teuthology.orchestra.run.smithi186.stdout:3 2024-05-04T17:44:21.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.280+0000 7f2787159640 10 librados: watch_flush enter 2024-05-04T17:44:21.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.280+0000 7f2787159640 10 librados: watch_flush exit 2024-05-04T17:44:21.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.280+0000 7f2787159640 20 client.4138.objecter shutdown clearing up homeless session... 2024-05-04T17:44:21.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.280+0000 7f2787159640 10 client.4138.objecter successfully canceled tick 2024-05-04T17:44:21.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.280+0000 7f2787159640 10 monclient: shutdown 2024-05-04T17:44:21.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.280+0000 7f2787159640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:21.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.280+0000 7f2787159640 1 -- 172.21.15.186:0/3418749626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27800b2e90 msgr2=0x7f27801443d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.280+0000 7f2787159640 1 --2- 172.21.15.186:0/3418749626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27800b2e90 0x7f27801443d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f277800b990 tx=0x7f277800be50 comp rx=0 tx=0).stop 2024-05-04T17:44:21.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.281+0000 7f2787159640 1 -- 172.21.15.186:0/3418749626 shutdown_connections 2024-05-04T17:44:21.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.281+0000 7f2787159640 1 --2- 172.21.15.186:0/3418749626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27800b2e90 0x7f27801443d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.281+0000 7f2787159640 1 --2- 172.21.15.186:0/3418749626 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f27800a9b00 0x7f2780143e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.281+0000 7f2787159640 1 --2- 172.21.15.186:0/3418749626 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f27800a9720 0x7f278013e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.281+0000 7f2787159640 1 -- 172.21.15.186:0/3418749626 >> 172.21.15.186:0/3418749626 conn(0x7f2780067780 msgr2=0x7f27800aa450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:21.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.281+0000 7f2787159640 1 -- 172.21.15.186:0/3418749626 shutdown_connections 2024-05-04T17:44:21.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.281+0000 7f2787159640 1 -- 172.21.15.186:0/3418749626 wait complete. 2024-05-04T17:44:21.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.281+0000 7f2787159640 1 librados: shutdown 2024-05-04T17:44:21.284 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:21.280+0000 7f9ef103e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-04T17:44:21.292 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new cb00f84f-d78c-461c-9843-73a9da4b80a4 4 2024-05-04T17:44:21.305 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:21.304+0000 7f33478fa040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-04T17:44:21.333 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:21.333+0000 7f9ef103e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-04T17:44:21.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.387+0000 7faac35bd640 10 monclient: get_monmap_and_config 2024-05-04T17:44:21.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.387+0000 7faac35bd640 10 monclient: build_initial_monmap 2024-05-04T17:44:21.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.387+0000 7faac35bd640 10 monclient: monmap: 2024-05-04T17:44:21.390 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:21.390 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.390 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:44:21.388758+0000 2024-05-04T17:44:21.390 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:44:21.388758+0000 2024-05-04T17:44:21.390 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:21.390 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.390 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:21.390 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:21.391 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:21.391 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:21.391 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:21.391 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.388+0000 7faac35bd640 1 Processor -- start 2024-05-04T17:44:21.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.388+0000 7faac35bd640 1 -- start start 2024-05-04T17:44:21.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.388+0000 7faac35bd640 10 monclient: init 2024-05-04T17:44:21.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:21.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faabc0a9720 0x7faabc0aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 10 monclient(hunting): picked mon.noname-c con 0x7faabc0a9720 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faabc0a9b00 addr v1:172.21.15.19:6789/0 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7faabc0b2e90 0x7faabc13fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 10 monclient(hunting): picked mon.noname-b con 0x7faabc0b2e90 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faabc0abc40 con 0x7faabc0a9b00 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7faabc146fa0 con 0x7faabc0a9720 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7faabc146090 con 0x7faabc0b2e90 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac35bd640 10 monclient(hunting): authenticate will time out at 1257.666855s 2024-05-04T17:44:21.394 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.389+0000 7faac1359640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faabc0a9720 0x7faabc0aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac0b58640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7faabc0a9b00 0x7faabc146890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.186:49182/0 (socket says 172.21.15.186:49182) 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac0b58640 1 -- 172.21.15.186:0/277520030 learned_addr learned my addr 172.21.15.186:0/277520030 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac1b5a640 1 --2- 172.21.15.186:0/277520030 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7faabc0b2e90 0x7faabc13fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac1359640 10 monclient(hunting): get_auth_request con 0x7faabc0a9720 auth_method 0 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac1359640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac1359640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac1359640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac1b5a640 10 monclient(hunting): get_auth_request con 0x7faabc0b2e90 auth_method 0 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac1b5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac1b5a640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac1b5a640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac1359640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac1359640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.390+0000 7faac1359640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.395 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.391+0000 7faac1b5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.391+0000 7faac1b5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.391+0000 7faac1b5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.392+0000 7faac1359640 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2024-05-04T17:44:21.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.392+0000 7faac1359640 1 -- 172.21.15.186:0/277520030 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7faabc0b2e90 msgr2=0x7faabc13fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:21.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.392+0000 7faac1359640 1 --2- 172.21.15.186:0/277520030 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7faabc0b2e90 0x7faabc13fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.392+0000 7faac1359640 1 -- 172.21.15.186:0/277520030 >> v1:172.21.15.19:6789/0 conn(0x7faabc0a9b00 legacy=0x7faabc146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.392+0000 7faac1359640 10 monclient: _finish_hunting 0 2024-05-04T17:44:21.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.392+0000 7faac1359640 1 monclient: found mon.noname-c 2024-05-04T17:44:21.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.392+0000 7faac1359640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:21.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.392+0000 7faac1359640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.392+0000 7faac1359640 1 -- 172.21.15.186:0/277520030 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faabc146370 con 0x7faabc0a9720 2024-05-04T17:44:21.397 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.394+0000 7faac1359640 1 --2- 172.21.15.186:0/277520030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faabc0a9720 0x7faabc0aa890 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faaac0097a0 tx=0x7faaac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d4467bdf6dd9d30 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:21.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 1 -- 172.21.15.186:0/277520030 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac035020 con 0x7faabc0a9720 2024-05-04T17:44:21.399 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 10 monclient: dump: 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 10 monclient: _finish_auth 0 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 10 monclient: _check_auth_tickets 2024-05-04T17:44:21.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:21.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 1 -- 172.21.15.186:0/277520030 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaac031830 con 0x7faabc0a9720 2024-05-04T17:44:21.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 5 monclient: authenticate success, global_id 4141 2024-05-04T17:44:21.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:44:21.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:21.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:21.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 1 -- 172.21.15.186:0/277520030 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac031c00 con 0x7faabc0a9720 2024-05-04T17:44:21.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 4 set_mon_vals no callback set 2024-05-04T17:44:21.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:21.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:21.401 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faab3fff640 10 monclient: dump: 2024-05-04T17:44:21.401 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:21.402 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.402 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.402 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.402 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:21.402 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.402 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:21.402 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:21.402 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:21.402 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 10 monclient: shutdown 2024-05-04T17:44:21.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:21.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 1 -- 172.21.15.186:0/277520030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faabc0a9720 msgr2=0x7faabc0aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 1 --2- 172.21.15.186:0/277520030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faabc0a9720 0x7faabc0aa890 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faaac0097a0 tx=0x7faaac02b8d0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 1 -- 172.21.15.186:0/277520030 shutdown_connections 2024-05-04T17:44:21.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 1 --2- 172.21.15.186:0/277520030 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7faabc0b2e90 0x7faabc13fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 1 --2- 172.21.15.186:0/277520030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faabc0a9720 0x7faabc0aa890 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 1 -- 172.21.15.186:0/277520030 >> 172.21.15.186:0/277520030 conn(0x7faabc067780 msgr2=0x7faabc0b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:21.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.398+0000 7faac35bd640 1 -- 172.21.15.186:0/277520030 shutdown_connections 2024-05-04T17:44:21.403 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.399+0000 7faac35bd640 1 -- 172.21.15.186:0/277520030 wait complete. 2024-05-04T17:44:21.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.399+0000 7faac35bd640 10 monclient: build_initial_monmap 2024-05-04T17:44:21.407 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.399+0000 7faac35bd640 10 monclient: monmap: 2024-05-04T17:44:21.407 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:21.407 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:21.407 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-05-04T17:44:21.407 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-05-04T17:44:21.407 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.400+0000 7faac35bd640 1 librados: starting msgr at 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.400+0000 7faac35bd640 1 librados: starting objecter 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.400+0000 7faac35bd640 1 Processor -- start 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.400+0000 7faac35bd640 1 -- start start 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.400+0000 7faac35bd640 1 librados: setting wanted keys 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.400+0000 7faac35bd640 1 librados: calling monclient init 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.400+0000 7faac35bd640 10 monclient: init 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-04T17:44:21.408 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7faabc0a9720 0x7faabc0b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 10 monclient(hunting): picked mon.noname-c con 0x7faabc0a9720 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faabc0a9b00 0x7faabc0acc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 10 monclient(hunting): picked mon.noname-b con 0x7faabc0a9b00 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faabc0b2e90 0x7faabc0ad800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 10 monclient(hunting): picked mon.noname-a con 0x7faabc0b2e90 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7faabc1407b0 con 0x7faabc0b2e90 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faabc140630 con 0x7faabc0a9b00 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7faabc14d420 con 0x7faabc0a9720 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.401+0000 7faac35bd640 10 monclient(hunting): authenticate will time out at 1257.679168s 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac0b58640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faabc0a9b00 0x7faabc0acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac1359640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7faabc0a9720 0x7faabc0b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac1b5a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faabc0b2e90 0x7faabc0ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac0b58640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faabc0a9b00 0x7faabc0acc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52764/0 (socket says 172.21.15.186:52764) 2024-05-04T17:44:21.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac0b58640 1 -- 172.21.15.186:0/1993122450 learned_addr learned my addr 172.21.15.186:0/1993122450 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:21.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac0b58640 10 monclient(hunting): get_auth_request con 0x7faabc0a9b00 auth_method 0 2024-05-04T17:44:21.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac0b58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac0b58640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac0b58640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac1b5a640 10 monclient(hunting): get_auth_request con 0x7faabc0b2e90 auth_method 0 2024-05-04T17:44:21.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac1b5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac1b5a640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac1b5a640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac1359640 10 monclient(hunting): get_auth_request con 0x7faabc0a9720 auth_method 0 2024-05-04T17:44:21.410 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac1359640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac1359640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac1359640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac0b58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac0b58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.414 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.402+0000 7faac0b58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.403+0000 7faac1b5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.403+0000 7faac1b5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.403+0000 7faac1b5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.403+0000 7faac1359640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.403+0000 7faac1359640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.403+0000 7faac1359640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.405+0000 7faac0b58640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.405+0000 7faac0b58640 1 -- 172.21.15.186:0/1993122450 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7faabc0a9720 msgr2=0x7faabc0b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.405+0000 7faac0b58640 1 --2- 172.21.15.186:0/1993122450 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7faabc0a9720 0x7faabc0b1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.405+0000 7faac0b58640 1 -- 172.21.15.186:0/1993122450 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faabc0b2e90 msgr2=0x7faabc0ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.405+0000 7faac0b58640 1 --2- 172.21.15.186:0/1993122450 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faabc0b2e90 0x7faabc0ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.405+0000 7faac0b58640 10 monclient: _finish_hunting 0 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.405+0000 7faac0b58640 1 monclient: found mon.noname-b 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.405+0000 7faac0b58640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:21.415 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.405+0000 7faac0b58640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.416 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.405+0000 7faac0b58640 1 -- 172.21.15.186:0/1993122450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faabc0ae930 con 0x7faabc0a9b00 2024-05-04T17:44:21.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.407+0000 7faac0b58640 1 --2- 172.21.15.186:0/1993122450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faabc0a9b00 0x7faabc0acc90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faab800c400 tx=0x7faab800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:21.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.407+0000 7faab1ffb640 10 client.?.objecter ms_handle_connect 0x7faabc0a9b00 2024-05-04T17:44:21.417 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.407+0000 7faab1ffb640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:21.418 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 1 -- 172.21.15.186:0/1993122450 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab8017070 con 0x7faabc0a9b00 2024-05-04T17:44:21.418 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:21.418 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:21.418 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 10 monclient: dump: 2024-05-04T17:44:21.418 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:21.418 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.418 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.418 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.418 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:21.418 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.418 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:21.418 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 10 monclient: _finish_auth 0 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 10 monclient: _check_auth_tickets 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faac35bd640 5 monclient: authenticate success, global_id 4144 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 1 -- 172.21.15.186:0/1993122450 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab8005ce0 con 0x7faabc0a9b00 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 1 -- 172.21.15.186:0/1993122450 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab80072f0 con 0x7faabc0a9b00 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faac35bd640 10 monclient: _renew_subs 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faac35bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab37fe640 4 set_mon_vals no callback set 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faac35bd640 1 -- 172.21.15.186:0/1993122450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faabc0aeba0 con 0x7faabc0a9b00 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:21.419 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:21.420 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faab1ffb640 10 monclient: dump: 2024-05-04T17:44:21.420 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:21.420 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.420 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.420 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.420 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:21.420 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.420 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:21.420 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:21.420 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:21.420 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faac35bd640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faac35bd640 10 monclient: _renew_subs 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faac35bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faac35bd640 1 -- 172.21.15.186:0/1993122450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faabc141ce0 con 0x7faabc0a9b00 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.411+0000 7faac35bd640 1 librados: init done 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.412+0000 7faac35bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.412+0000 7faac35bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.412+0000 7faac35bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.412+0000 7faac35bd640 1 -- 172.21.15.186:0/1993122450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faabc06a700 con 0x7faabc0a9b00 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.414+0000 7faab1ffb640 1 -- 172.21.15.186:0/1993122450 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7faab801a070 con 0x7faabc0a9b00 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.414+0000 7faab1ffb640 1 -- 172.21.15.186:0/1993122450 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7faab801b8c0 con 0x7faabc0a9b00 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.414+0000 7faab1ffb640 10 client.4144.objecter ms_dispatch 0x7faabc000b90 osd_map(6..6 src has 1..6) 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.414+0000 7faab1ffb640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.414+0000 7faab1ffb640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-05-04T17:44:21.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.414+0000 7faab1ffb640 20 client.4144.objecter dump_active .. 0 homeless 2024-05-04T17:44:21.471 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:21.471+0000 7f33478fa040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-04T17:44:21.499 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:21.499+0000 7f9ef103e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-04T17:44:21.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.514+0000 7faab1ffb640 1 -- 172.21.15.186:0/1993122450 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7faab8021020 con 0x7faabc0a9b00 2024-05-04T17:44:21.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.514+0000 7faab1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:21.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.514+0000 7faab1ffb640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:21.588 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.586+0000 7faac35bd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cb00f84f-d78c-461c-9843-73a9da4b80a4", "id": 4}] 2024-05-04T17:44:21.588 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.586+0000 7faac35bd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cb00f84f-d78c-461c-9843-73a9da4b80a4", "id": 4}] 2024-05-04T17:44:21.588 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.587+0000 7faac35bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.588 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.587+0000 7faac35bd640 1 -- 172.21.15.186:0/1993122450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cb00f84f-d78c-461c-9843-73a9da4b80a4", "id": 4} v 0) -- 0x7faabc13bf10 con 0x7faabc0a9b00 2024-05-04T17:44:21.613 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:21.612+0000 7f33478fa040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-04T17:44:21.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.658+0000 7faab1ffb640 1 -- 172.21.15.186:0/1993122450 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cb00f84f-d78c-461c-9843-73a9da4b80a4", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7faab8007490 con 0x7faabc0a9b00 2024-05-04T17:44:21.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.658+0000 7faab1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cb00f84f-d78c-461c-9843-73a9da4b80a4", "id": 4}] 2024-05-04T17:44:21.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.658+0000 7faab1ffb640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:21.660 INFO:teuthology.orchestra.run.smithi186.stdout:4 2024-05-04T17:44:21.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.659+0000 7faac35bd640 10 librados: watch_flush enter 2024-05-04T17:44:21.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.659+0000 7faac35bd640 10 librados: watch_flush exit 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.659+0000 7faac35bd640 20 client.4144.objecter shutdown clearing up homeless session... 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.659+0000 7faac35bd640 10 client.4144.objecter successfully canceled tick 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.659+0000 7faac35bd640 10 monclient: shutdown 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.659+0000 7faac35bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.659+0000 7faac35bd640 1 -- 172.21.15.186:0/1993122450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faabc0a9b00 msgr2=0x7faabc0acc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.659+0000 7faac35bd640 1 --2- 172.21.15.186:0/1993122450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faabc0a9b00 0x7faabc0acc90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7faab800c400 tx=0x7faab800c8c0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.659+0000 7faac35bd640 1 -- 172.21.15.186:0/1993122450 shutdown_connections 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.660+0000 7faac35bd640 1 --2- 172.21.15.186:0/1993122450 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7faabc0b2e90 0x7faabc0ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.660+0000 7faac35bd640 1 --2- 172.21.15.186:0/1993122450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faabc0a9b00 0x7faabc0acc90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.660+0000 7faac35bd640 1 --2- 172.21.15.186:0/1993122450 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7faabc0a9720 0x7faabc0b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.660+0000 7faac35bd640 1 -- 172.21.15.186:0/1993122450 >> 172.21.15.186:0/1993122450 conn(0x7faabc067780 msgr2=0x7faabc13b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.660+0000 7faac35bd640 1 -- 172.21.15.186:0/1993122450 shutdown_connections 2024-05-04T17:44:21.662 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.660+0000 7faac35bd640 1 -- 172.21.15.186:0/1993122450 wait complete. 2024-05-04T17:44:21.663 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.660+0000 7faac35bd640 1 librados: shutdown 2024-05-04T17:44:21.663 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:21.660+0000 7f9ef103e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-04T17:44:21.666 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:21.666+0000 7f33478fa040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-04T17:44:21.671 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new d07ad403-291f-4bde-a64a-5181fb79e544 5 2024-05-04T17:44:21.715 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:21.714+0000 7f9ef103e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-04T17:44:21.740 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:21.740+0000 7f33478fa040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-04T17:44:21.788 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:21.787+0000 7f33478fa040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-04T17:44:21.789 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:21.787+0000 7f9ef103e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-04T17:44:21.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.807+0000 7ff54cb80640 10 monclient: get_monmap_and_config 2024-05-04T17:44:21.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.807+0000 7ff54cb80640 10 monclient: build_initial_monmap 2024-05-04T17:44:21.808 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.807+0000 7ff54cb80640 10 monclient: monmap: 2024-05-04T17:44:21.808 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:21.808 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.809 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:44:21.808617+0000 2024-05-04T17:44:21.809 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:44:21.808617+0000 2024-05-04T17:44:21.809 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:21.809 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.809 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:21.809 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:21.809 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:21.809 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:21.809 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:21.809 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.813 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.808+0000 7ff54cb80640 1 Processor -- start 2024-05-04T17:44:21.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.808+0000 7ff54cb80640 1 -- start start 2024-05-04T17:44:21.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.808+0000 7ff54cb80640 10 monclient: init 2024-05-04T17:44:21.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:21.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-05-04T17:44:21.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5480a9fb0 0x7ff5480aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 10 monclient(hunting): picked mon.noname-b con 0x7ff5480a9fb0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:21.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff5480b27f0 0x7ff54813fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 10 monclient(hunting): picked mon.noname-c con 0x7ff5480b27f0 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:21.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7ff548140310 0x7ff5481463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 10 monclient(hunting): picked mon.noname-a con 0x7ff548140310 addr v2:172.21.15.19:3300/0 2024-05-04T17:44:21.814 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7ff5481407a0 con 0x7ff548140310 2024-05-04T17:44:21.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff5480ab880 con 0x7ff5480b27f0 2024-05-04T17:44:21.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7ff548146930 con 0x7ff5480a9fb0 2024-05-04T17:44:21.815 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:21.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54cb80640 10 monclient(hunting): authenticate will time out at 1258.086710s 2024-05-04T17:44:21.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff545d9b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff5480b27f0 0x7ff54813fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff54659c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5480a9fb0 0x7ff5480aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.809+0000 7ff546d9d640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7ff548140310 0x7ff5481463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff545d9b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff5480b27f0 0x7ff54813fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52770/0 (socket says 172.21.15.186:52770) 2024-05-04T17:44:21.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff545d9b640 1 -- 172.21.15.186:0/1839127670 learned_addr learned my addr 172.21.15.186:0/1839127670 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:21.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff545d9b640 10 monclient(hunting): get_auth_request con 0x7ff5480b27f0 auth_method 0 2024-05-04T17:44:21.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff545d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff545d9b640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff545d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff546d9d640 10 monclient(hunting): get_auth_request con 0x7ff548140310 auth_method 0 2024-05-04T17:44:21.816 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff546d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff546d9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff546d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff54659c640 10 monclient(hunting): get_auth_request con 0x7ff5480a9fb0 auth_method 0 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff54659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff54659c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff54659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff545d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff545d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff545d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff546d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff546d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff546d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff54659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff54659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.810+0000 7ff54659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.812+0000 7ff545d9b640 10 monclient(hunting): handle_auth_done global_id 4147 payload 274 2024-05-04T17:44:21.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.812+0000 7ff545d9b640 1 -- 172.21.15.186:0/1839127670 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5480a9fb0 msgr2=0x7ff5480aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:21.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.812+0000 7ff545d9b640 1 --2- 172.21.15.186:0/1839127670 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5480a9fb0 0x7ff5480aa390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.812+0000 7ff545d9b640 1 -- 172.21.15.186:0/1839127670 >> v2:172.21.15.19:3300/0 conn(0x7ff548140310 msgr2=0x7ff5481463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:21.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.812+0000 7ff545d9b640 1 --2- 172.21.15.186:0/1839127670 >> v2:172.21.15.19:3300/0 conn(0x7ff548140310 0x7ff5481463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.812+0000 7ff545d9b640 10 monclient: _finish_hunting 0 2024-05-04T17:44:21.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.812+0000 7ff545d9b640 1 monclient: found mon.noname-c 2024-05-04T17:44:21.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.812+0000 7ff545d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:21.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.812+0000 7ff545d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.818 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.812+0000 7ff545d9b640 1 -- 172.21.15.186:0/1839127670 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff548146ae0 con 0x7ff5480b27f0 2024-05-04T17:44:21.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.814+0000 7ff545d9b640 1 --2- 172.21.15.186:0/1839127670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5480b27f0 0x7ff54813fdd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff5480730f0 tx=0x7ff53c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5b1e050cb37b61a server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:21.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.817+0000 7ff54559a640 1 -- 172.21.15.186:0/1839127670 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c02e070 con 0x7ff5480b27f0 2024-05-04T17:44:21.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.817+0000 7ff54559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:21.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.817+0000 7ff54559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:44:21.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54559a640 10 monclient: dump: 2024-05-04T17:44:21.827 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:21.827 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.827 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54559a640 10 monclient: _finish_auth 0 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54559a640 10 monclient: _check_auth_tickets 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54559a640 1 -- 172.21.15.186:0/1839127670 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff53c037070 con 0x7ff5480b27f0 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 5 monclient: authenticate success, global_id 4147 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54559a640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:21.828 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54559a640 1 -- 172.21.15.186:0/1839127670 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c031a00 con 0x7ff5480b27f0 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 4 set_mon_vals no callback set 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54559a640 10 monclient: dump: 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:21.829 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 10 monclient: shutdown 2024-05-04T17:44:21.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:21.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 1 -- 172.21.15.186:0/1839127670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5480b27f0 msgr2=0x7ff54813fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 1 --2- 172.21.15.186:0/1839127670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5480b27f0 0x7ff54813fdd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff5480730f0 tx=0x7ff53c02b880 comp rx=0 tx=0).stop 2024-05-04T17:44:21.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 1 -- 172.21.15.186:0/1839127670 shutdown_connections 2024-05-04T17:44:21.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 1 --2- 172.21.15.186:0/1839127670 >> v2:172.21.15.19:3300/0 conn(0x7ff548140310 0x7ff5481463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 1 --2- 172.21.15.186:0/1839127670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5480b27f0 0x7ff54813fdd0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 1 --2- 172.21.15.186:0/1839127670 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5480a9fb0 0x7ff5480aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 1 -- 172.21.15.186:0/1839127670 >> 172.21.15.186:0/1839127670 conn(0x7ff5480b4e90 msgr2=0x7ff5480b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:21.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 1 -- 172.21.15.186:0/1839127670 shutdown_connections 2024-05-04T17:44:21.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.818+0000 7ff54cb80640 1 -- 172.21.15.186:0/1839127670 wait complete. 2024-05-04T17:44:21.830 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.819+0000 7ff54cb80640 10 monclient: build_initial_monmap 2024-05-04T17:44:21.831 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 10 monclient: monmap: 2024-05-04T17:44:21.831 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:21.831 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:21.831 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-05-04T17:44:21.831 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-05-04T17:44:21.831 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:21.831 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.831 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:21.832 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:21.832 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:21.832 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 1 librados: starting msgr at 2024-05-04T17:44:21.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 1 librados: starting objecter 2024-05-04T17:44:21.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 1 Processor -- start 2024-05-04T17:44:21.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 1 -- start start 2024-05-04T17:44:21.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 1 librados: setting wanted keys 2024-05-04T17:44:21.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 1 librados: calling monclient init 2024-05-04T17:44:21.832 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 10 monclient: init 2024-05-04T17:44:21.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:21.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-04T17:44:21.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5480a9fb0 0x7ff5480ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 10 monclient(hunting): picked mon.noname-c con 0x7ff5480a9fb0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:21.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5480b27f0 0x7ff5481389a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 10 monclient(hunting): picked mon.noname-b con 0x7ff5480b27f0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:21.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff548140310 0x7ff548138ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:21.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 10 monclient(hunting): picked mon.noname-a con 0x7ff548140310 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:21.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff548142ce0 con 0x7ff548140310 2024-05-04T17:44:21.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.833 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff548142b60 con 0x7ff5480b27f0 2024-05-04T17:44:21.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:21.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7ff548142e60 con 0x7ff5480a9fb0 2024-05-04T17:44:21.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:21.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.820+0000 7ff54cb80640 10 monclient(hunting): authenticate will time out at 1258.098528s 2024-05-04T17:44:21.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff545d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5480b27f0 0x7ff5481389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff54659c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5480a9fb0 0x7ff5480ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff546d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff548140310 0x7ff548138ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:21.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff546d9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff548140310 0x7ff548138ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.186:44984/0 (socket says 172.21.15.186:44984) 2024-05-04T17:44:21.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff54659c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5480a9fb0 0x7ff5480ac1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.186:57568/0 (socket says 172.21.15.186:57568) 2024-05-04T17:44:21.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff546d9d640 1 -- 172.21.15.186:0/197777929 learned_addr learned my addr 172.21.15.186:0/197777929 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:21.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff546d9d640 10 monclient(hunting): get_auth_request con 0x7ff548140310 auth_method 0 2024-05-04T17:44:21.834 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff546d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff546d9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff546d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff54659c640 10 monclient(hunting): get_auth_request con 0x7ff5480a9fb0 auth_method 0 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff54659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff54659c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff54659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff545d9b640 10 monclient(hunting): get_auth_request con 0x7ff5480b27f0 auth_method 0 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff545d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff545d9b640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.821+0000 7ff545d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.822+0000 7ff546d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.822+0000 7ff546d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.822+0000 7ff546d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.822+0000 7ff545d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.835 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.822+0000 7ff545d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.822+0000 7ff545d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.822+0000 7ff54659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:21.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.822+0000 7ff54659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:21.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.822+0000 7ff54659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:21.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.825+0000 7ff545d9b640 10 monclient(hunting): handle_auth_done global_id 4150 payload 824 2024-05-04T17:44:21.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.825+0000 7ff545d9b640 1 -- 172.21.15.186:0/197777929 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5480a9fb0 msgr2=0x7ff5480ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.825+0000 7ff545d9b640 1 --2- 172.21.15.186:0/197777929 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5480a9fb0 0x7ff5480ac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.825+0000 7ff545d9b640 1 -- 172.21.15.186:0/197777929 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff548140310 msgr2=0x7ff548138ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:21.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.825+0000 7ff545d9b640 1 --2- 172.21.15.186:0/197777929 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff548140310 0x7ff548138ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:21.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.825+0000 7ff545d9b640 10 monclient: _finish_hunting 0 2024-05-04T17:44:21.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.825+0000 7ff545d9b640 1 monclient: found mon.noname-b 2024-05-04T17:44:21.836 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.825+0000 7ff545d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:21.837 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.825+0000 7ff545d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.837 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.825+0000 7ff545d9b640 1 -- 172.21.15.186:0/197777929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5480ae0e0 con 0x7ff5480b27f0 2024-05-04T17:44:21.837 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.825+0000 7ff546d9d640 1 --2- 172.21.15.186:0/197777929 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff548140310 0x7ff548138ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-04T17:44:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.827+0000 7ff545d9b640 1 --2- 172.21.15.186:0/197777929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5480b27f0 0x7ff5481389a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff5480aba00 tx=0x7ff53c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.827+0000 7ff536ffd640 10 client.?.objecter ms_handle_connect 0x7ff5480b27f0 2024-05-04T17:44:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.827+0000 7ff536ffd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 1 -- 172.21.15.186:0/197777929 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c032670 con 0x7ff5480b27f0 2024-05-04T17:44:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 10 monclient: dump: 2024-05-04T17:44:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.839 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 10 monclient: _finish_auth 0 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 10 monclient: _check_auth_tickets 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 1 -- 172.21.15.186:0/197777929 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff53c032830 con 0x7ff5480b27f0 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 1 -- 172.21.15.186:0/197777929 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff53c032c20 con 0x7ff5480b27f0 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff54cb80640 5 monclient: authenticate success, global_id 4150 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff54cb80640 10 monclient: _renew_subs 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff54cb80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff54cb80640 1 -- 172.21.15.186:0/197777929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5480ae3c0 con 0x7ff5480b27f0 2024-05-04T17:44:21.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff544d99640 4 set_mon_vals no callback set 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff536ffd640 10 monclient: dump: 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff54cb80640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:21.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff54cb80640 10 monclient: _renew_subs 2024-05-04T17:44:21.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff54cb80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff54cb80640 1 -- 172.21.15.186:0/197777929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5480acb10 con 0x7ff5480b27f0 2024-05-04T17:44:21.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.831+0000 7ff54cb80640 1 librados: init done 2024-05-04T17:44:21.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.832+0000 7ff54cb80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:21.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.832+0000 7ff54cb80640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:21.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.832+0000 7ff54cb80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:21.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.832+0000 7ff54cb80640 1 -- 172.21.15.186:0/197777929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff548068b10 con 0x7ff5480b27f0 2024-05-04T17:44:21.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.835+0000 7ff536ffd640 1 -- 172.21.15.186:0/197777929 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff53c04c060 con 0x7ff5480b27f0 2024-05-04T17:44:21.843 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:21.838+0000 7f9ef103e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-04T17:44:21.938 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.937+0000 7ff536ffd640 1 -- 172.21.15.186:0/197777929 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7ff53c006000 con 0x7ff5480b27f0 2024-05-04T17:44:21.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.937+0000 7ff536ffd640 10 client.4150.objecter ms_dispatch 0x7ff548000b90 osd_map(7..7 src has 1..7) 2024-05-04T17:44:21.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.937+0000 7ff536ffd640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2024-05-04T17:44:21.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.937+0000 7ff536ffd640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2024-05-04T17:44:21.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.937+0000 7ff536ffd640 20 client.4150.objecter dump_active .. 0 homeless 2024-05-04T17:44:21.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.937+0000 7ff536ffd640 1 -- 172.21.15.186:0/197777929 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff5480ae0e0 con 0x7ff5480b27f0 2024-05-04T17:44:21.939 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.937+0000 7ff536ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:21.940 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:21.937+0000 7ff536ffd640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:22.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.010+0000 7ff54cb80640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d07ad403-291f-4bde-a64a-5181fb79e544", "id": 5}] 2024-05-04T17:44:22.011 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.010+0000 7ff54cb80640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d07ad403-291f-4bde-a64a-5181fb79e544", "id": 5}] 2024-05-04T17:44:22.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.010+0000 7ff54cb80640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:22.012 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.010+0000 7ff54cb80640 1 -- 172.21.15.186:0/197777929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d07ad403-291f-4bde-a64a-5181fb79e544", "id": 5} v 0) -- 0x7ff548139420 con 0x7ff5480b27f0 2024-05-04T17:44:22.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.080+0000 7ff536ffd640 1 -- 172.21.15.186:0/197777929 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d07ad403-291f-4bde-a64a-5181fb79e544", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7ff53c040070 con 0x7ff5480b27f0 2024-05-04T17:44:22.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.080+0000 7ff536ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d07ad403-291f-4bde-a64a-5181fb79e544", "id": 5}] 2024-05-04T17:44:22.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.080+0000 7ff536ffd640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:22.082 INFO:teuthology.orchestra.run.smithi186.stdout:5 2024-05-04T17:44:22.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 10 librados: watch_flush enter 2024-05-04T17:44:22.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 10 librados: watch_flush exit 2024-05-04T17:44:22.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 20 client.4150.objecter shutdown clearing up homeless session... 2024-05-04T17:44:22.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 10 client.4150.objecter successfully canceled tick 2024-05-04T17:44:22.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 10 monclient: shutdown 2024-05-04T17:44:22.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:22.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 1 -- 172.21.15.186:0/197777929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5480b27f0 msgr2=0x7ff5481389a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:22.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 1 --2- 172.21.15.186:0/197777929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5480b27f0 0x7ff5481389a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff5480aba00 tx=0x7ff53c0046c0 comp rx=0 tx=0).stop 2024-05-04T17:44:22.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 1 -- 172.21.15.186:0/197777929 shutdown_connections 2024-05-04T17:44:22.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 1 --2- 172.21.15.186:0/197777929 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff548140310 0x7ff548138ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:22.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 1 --2- 172.21.15.186:0/197777929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5480b27f0 0x7ff5481389a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:22.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 1 --2- 172.21.15.186:0/197777929 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5480a9fb0 0x7ff5480ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:22.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 1 -- 172.21.15.186:0/197777929 >> 172.21.15.186:0/197777929 conn(0x7ff5480b4e90 msgr2=0x7ff5481461b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:22.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 1 -- 172.21.15.186:0/197777929 shutdown_connections 2024-05-04T17:44:22.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 1 -- 172.21.15.186:0/197777929 wait complete. 2024-05-04T17:44:22.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.081+0000 7ff54cb80640 1 librados: shutdown 2024-05-04T17:44:22.093 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 5110d1ad-5634-4c8e-9338-d6c4229b1d63 6 2024-05-04T17:44:22.239 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.231+0000 7f6f7b0ba640 10 monclient: get_monmap_and_config 2024-05-04T17:44:22.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.231+0000 7f6f7b0ba640 10 monclient: build_initial_monmap 2024-05-04T17:44:22.240 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.231+0000 7f6f7b0ba640 10 monclient: monmap: 2024-05-04T17:44:22.240 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:22.240 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:22.240 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:44:22.232106+0000 2024-05-04T17:44:22.241 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:44:22.232106+0000 2024-05-04T17:44:22.241 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:22.241 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:22.241 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:22.241 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:22.241 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:22.241 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:22.242 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:22.242 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:22.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.231+0000 7f6f7b0ba640 1 Processor -- start 2024-05-04T17:44:22.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.231+0000 7f6f7b0ba640 1 -- start start 2024-05-04T17:44:22.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 10 monclient: init 2024-05-04T17:44:22.242 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:22.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-05-04T17:44:22.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f740b37a0 addr v1:172.21.15.19:6789/0 2024-05-04T17:44:22.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6f74072c50 0x7f6f740b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:22.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 10 monclient(hunting): picked mon.noname-b con 0x7f6f74072c50 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:22.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f740b2270 addr v1:172.21.15.186:6789/0 2024-05-04T17:44:22.243 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f740a9790 con 0x7f6f740b37a0 2024-05-04T17:44:22.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f740a9a70 con 0x7f6f740b2270 2024-05-04T17:44:22.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:22.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f6f740b3b80 con 0x7f6f74072c50 2024-05-04T17:44:22.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:22.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.232+0000 7f6f7b0ba640 10 monclient(hunting): authenticate will time out at 1258.510184s 2024-05-04T17:44:22.244 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.233+0000 7f6f79657640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6f740b2270 0x7f6f7414b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:37316/0 (socket says 172.21.15.186:37316) 2024-05-04T17:44:22.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.233+0000 7f6f79657640 1 -- 172.21.15.186:0/1230953037 learned_addr learned my addr 172.21.15.186:0/1230953037 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-05-04T17:44:22.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.233+0000 7f6f73fff640 1 --2- 172.21.15.186:0/1230953037 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6f74072c50 0x7f6f740b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:22.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.233+0000 7f6f73fff640 10 monclient(hunting): get_auth_request con 0x7f6f74072c50 auth_method 0 2024-05-04T17:44:22.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.233+0000 7f6f73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:22.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.233+0000 7f6f73fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:22.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.233+0000 7f6f73fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:22.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.234+0000 7f6f73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:22.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.234+0000 7f6f73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:22.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.234+0000 7f6f73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:22.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.236+0000 7f6f73fff640 10 monclient(hunting): handle_auth_done global_id 4139 payload 274 2024-05-04T17:44:22.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.236+0000 7f6f73fff640 1 -- 172.21.15.186:0/1230953037 >> v1:172.21.15.186:6789/0 conn(0x7f6f740b2270 legacy=0x7f6f7414b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:22.246 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.236+0000 7f6f73fff640 1 -- 172.21.15.186:0/1230953037 >> v1:172.21.15.19:6789/0 conn(0x7f6f740b37a0 legacy=0x7f6f7413dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:22.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.236+0000 7f6f73fff640 10 monclient: _finish_hunting 0 2024-05-04T17:44:22.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.236+0000 7f6f73fff640 1 monclient: found mon.noname-b 2024-05-04T17:44:22.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.236+0000 7f6f73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:22.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.236+0000 7f6f73fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:44:22.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.236+0000 7f6f73fff640 1 -- 172.21.15.186:0/1230953037 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f74071af0 con 0x7f6f74072c50 2024-05-04T17:44:22.247 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.238+0000 7f6f73fff640 1 --2- 172.21.15.186:0/1230953037 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6f74072c50 0x7f6f740b1d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6f600097d0 tx=0x7f6f6002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf6f3665deff50c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:22.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 1 -- 172.21.15.186:0/1230953037 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60035020 con 0x7f6f74072c50 2024-05-04T17:44:22.260 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:22.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:22.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 10 monclient: dump: 2024-05-04T17:44:22.261 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:22.261 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:22.261 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:22.261 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:22.262 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:22.262 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:22.262 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:22.262 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:22.262 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:22.263 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:22.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 10 monclient: _finish_auth 0 2024-05-04T17:44:22.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 10 monclient: _check_auth_tickets 2024-05-04T17:44:22.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:22.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 1 -- 172.21.15.186:0/1230953037 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f60031800 con 0x7f6f74072c50 2024-05-04T17:44:22.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:22.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 1 -- 172.21.15.186:0/1230953037 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f60031bd0 con 0x7f6f74072c50 2024-05-04T17:44:22.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f7b0ba640 5 monclient: authenticate success, global_id 4139 2024-05-04T17:44:22.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f7b0ba640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:22.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f7b0ba640 4 set_mon_vals no callback set 2024-05-04T17:44:22.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:22.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:44:22.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f737fe640 10 monclient: dump: 2024-05-04T17:44:22.265 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:22.265 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:22.265 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:22.265 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:22.265 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:22.266 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:22.266 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:22.266 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:22.266 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:22.266 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:22.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f7b0ba640 10 monclient: shutdown 2024-05-04T17:44:22.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f7b0ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:22.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1230953037 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6f74072c50 msgr2=0x7f6f740b1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:22.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f7b0ba640 1 --2- 172.21.15.186:0/1230953037 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6f74072c50 0x7f6f740b1d30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6f600097d0 tx=0x7f6f6002b8a0 comp rx=0 tx=0).stop 2024-05-04T17:44:22.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1230953037 shutdown_connections 2024-05-04T17:44:22.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f7b0ba640 1 --2- 172.21.15.186:0/1230953037 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6f74072c50 0x7f6f740b1d30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:22.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.242+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1230953037 >> 172.21.15.186:0/1230953037 conn(0x7f6f740b6950 msgr2=0x7f6f740b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:22.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.243+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1230953037 shutdown_connections 2024-05-04T17:44:22.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.243+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1230953037 wait complete. 2024-05-04T17:44:22.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.243+0000 7f6f7b0ba640 10 monclient: build_initial_monmap 2024-05-04T17:44:22.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.243+0000 7f6f7b0ba640 10 monclient: monmap: 2024-05-04T17:44:22.268 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:22.268 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:22.268 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-05-04T17:44:22.268 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-05-04T17:44:22.269 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:22.269 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:22.269 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:22.269 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:22.269 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:22.269 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:22.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.243+0000 7f6f7b0ba640 1 librados: starting msgr at 2024-05-04T17:44:22.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.243+0000 7f6f7b0ba640 1 librados: starting objecter 2024-05-04T17:44:22.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.243+0000 7f6f7b0ba640 1 Processor -- start 2024-05-04T17:44:22.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 1 -- start start 2024-05-04T17:44:22.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 1 librados: setting wanted keys 2024-05-04T17:44:22.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 1 librados: calling monclient init 2024-05-04T17:44:22.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 10 monclient: init 2024-05-04T17:44:22.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:22.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:44:22.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6f740b2270 0x7f6f741d7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:22.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 10 monclient(hunting): picked mon.noname-a con 0x7f6f740b2270 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:22.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6f740b37a0 0x7f6f741d7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:22.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 10 monclient(hunting): picked mon.noname-c con 0x7f6f740b37a0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:22.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f741391d0 0x7f6f741c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:22.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 10 monclient(hunting): picked mon.noname-b con 0x7f6f741391d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:22.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:22.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6f7413e3d0 con 0x7f6f740b2270 2024-05-04T17:44:22.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:22.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6f740b2670 con 0x7f6f741391d0 2024-05-04T17:44:22.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:22.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f6f740aac90 con 0x7f6f740b37a0 2024-05-04T17:44:22.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:22.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.244+0000 7f6f7b0ba640 10 monclient(hunting): authenticate will time out at 1258.522356s 2024-05-04T17:44:22.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f78e56640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6f740b2270 0x7f6f741d7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:22.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f79657640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f741391d0 0x7f6f741c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:22.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f73fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6f740b37a0 0x7f6f741d7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:22.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f73fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6f740b37a0 0x7f6f741d7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.186:57572/0 (socket says 172.21.15.186:57572) 2024-05-04T17:44:22.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f73fff640 1 -- 172.21.15.186:0/1231123255 learned_addr learned my addr 172.21.15.186:0/1231123255 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:22.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f73fff640 10 monclient(hunting): get_auth_request con 0x7f6f740b37a0 auth_method 0 2024-05-04T17:44:22.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:22.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f73fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:22.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f73fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:22.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f78e56640 10 monclient(hunting): get_auth_request con 0x7f6f740b2270 auth_method 0 2024-05-04T17:44:22.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f78e56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:22.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f78e56640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:22.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f78e56640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:22.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f79657640 10 monclient(hunting): get_auth_request con 0x7f6f741391d0 auth_method 0 2024-05-04T17:44:22.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f79657640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:22.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f79657640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:22.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.245+0000 7f6f79657640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:22.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.246+0000 7f6f73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:22.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.246+0000 7f6f73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:22.276 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.246+0000 7f6f73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:22.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.246+0000 7f6f78e56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:22.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.246+0000 7f6f78e56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:22.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.246+0000 7f6f78e56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:22.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.246+0000 7f6f79657640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:22.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.246+0000 7f6f79657640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:22.277 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.246+0000 7f6f79657640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:22.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.249+0000 7f6f79657640 10 monclient(hunting): handle_auth_done global_id 4156 payload 824 2024-05-04T17:44:22.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.249+0000 7f6f79657640 1 -- 172.21.15.186:0/1231123255 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6f740b37a0 msgr2=0x7f6f741d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:22.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.249+0000 7f6f79657640 1 --2- 172.21.15.186:0/1231123255 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6f740b37a0 0x7f6f741d7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:22.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.249+0000 7f6f79657640 1 -- 172.21.15.186:0/1231123255 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6f740b2270 msgr2=0x7f6f741d7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:22.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.249+0000 7f6f79657640 1 --2- 172.21.15.186:0/1231123255 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6f740b2270 0x7f6f741d7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:22.278 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.249+0000 7f6f79657640 10 monclient: _finish_hunting 0 2024-05-04T17:44:22.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.249+0000 7f6f79657640 1 monclient: found mon.noname-b 2024-05-04T17:44:22.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.249+0000 7f6f79657640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:22.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.249+0000 7f6f79657640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:22.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.249+0000 7f6f79657640 1 -- 172.21.15.186:0/1231123255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f741c6f30 con 0x7f6f741391d0 2024-05-04T17:44:22.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.251+0000 7f6f79657640 1 --2- 172.21.15.186:0/1231123255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f741391d0 0x7f6f741c69f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00caa0 tx=0x7f6f6c00cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:22.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.251+0000 7f6f717fa640 10 client.?.objecter ms_handle_connect 0x7f6f741391d0 2024-05-04T17:44:22.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.251+0000 7f6f717fa640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:22.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 1 -- 172.21.15.186:0/1231123255 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c017070 con 0x7f6f741391d0 2024-05-04T17:44:22.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:22.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:22.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 10 monclient: dump: 2024-05-04T17:44:22.280 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:22.281 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:22.281 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:22.281 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:22.281 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:22.281 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:22.281 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:22.282 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:22.282 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:22.282 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:22.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 10 monclient: _finish_auth 0 2024-05-04T17:44:22.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 10 monclient: _check_auth_tickets 2024-05-04T17:44:22.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:22.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 1 -- 172.21.15.186:0/1231123255 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f6c00ac00 con 0x7f6f741391d0 2024-05-04T17:44:22.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f7b0ba640 5 monclient: authenticate success, global_id 4156 2024-05-04T17:44:22.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f7b0ba640 10 monclient: _renew_subs 2024-05-04T17:44:22.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f7b0ba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:22.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1231123255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f741c7160 con 0x7f6f741391d0 2024-05-04T17:44:22.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:22.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 1 -- 172.21.15.186:0/1231123255 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c012400 con 0x7f6f741391d0 2024-05-04T17:44:22.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:22.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:22.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f717fa640 10 monclient: dump: 2024-05-04T17:44:22.284 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:22.284 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:22.285 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:22.285 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:22.285 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:22.285 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:22.285 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:22.285 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:22.285 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:22.286 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:22.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.254+0000 7f6f72ffd640 4 set_mon_vals no callback set 2024-05-04T17:44:22.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.255+0000 7f6f7b0ba640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:22.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.255+0000 7f6f7b0ba640 10 monclient: _renew_subs 2024-05-04T17:44:22.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.255+0000 7f6f7b0ba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:22.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.255+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1231123255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f740af8e0 con 0x7f6f741391d0 2024-05-04T17:44:22.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.255+0000 7f6f7b0ba640 1 librados: init done 2024-05-04T17:44:22.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.256+0000 7f6f7b0ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:22.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.256+0000 7f6f7b0ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:22.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.256+0000 7f6f7b0ba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:22.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.256+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1231123255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f740b1d30 con 0x7f6f741391d0 2024-05-04T17:44:22.293 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:22.292+0000 7f33478fa040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-04T17:44:22.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.362+0000 7f6f717fa640 1 -- 172.21.15.186:0/1231123255 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6f6c01a070 con 0x7f6f741391d0 2024-05-04T17:44:22.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.362+0000 7f6f717fa640 1 -- 172.21.15.186:0/1231123255 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f6f6c007660 con 0x7f6f741391d0 2024-05-04T17:44:22.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.362+0000 7f6f717fa640 10 client.4156.objecter ms_dispatch 0x7f6f74000b90 osd_map(8..8 src has 1..8) 2024-05-04T17:44:22.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.362+0000 7f6f717fa640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2024-05-04T17:44:22.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.362+0000 7f6f717fa640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2024-05-04T17:44:22.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.362+0000 7f6f717fa640 20 client.4156.objecter dump_active .. 0 homeless 2024-05-04T17:44:22.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.362+0000 7f6f717fa640 1 -- 172.21.15.186:0/1231123255 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6f740b1d30 con 0x7f6f741391d0 2024-05-04T17:44:22.364 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.362+0000 7f6f717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:22.365 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.362+0000 7f6f717fa640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:22.371 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:22.370+0000 7f9ef103e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-04T17:44:22.417 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:22.416+0000 7f33478fa040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-04T17:44:22.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.435+0000 7f6f7b0ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5110d1ad-5634-4c8e-9338-d6c4229b1d63", "id": 6}] 2024-05-04T17:44:22.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.435+0000 7f6f7b0ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5110d1ad-5634-4c8e-9338-d6c4229b1d63", "id": 6}] 2024-05-04T17:44:22.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.435+0000 7f6f7b0ba640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:22.438 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.435+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1231123255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5110d1ad-5634-4c8e-9338-d6c4229b1d63", "id": 6} v 0) -- 0x7f6f7406a700 con 0x7f6f741391d0 2024-05-04T17:44:22.491 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-05-04T17:44:22.490+0000 7f9ef103e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-04T17:44:22.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.847+0000 7f6f717fa640 1 -- 172.21.15.186:0/1231123255 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mgrmap(e 2) ==== 45274+0+0 (secure 0 0 0) 0x7f6f6c007070 con 0x7f6f741391d0 2024-05-04T17:44:22.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.962+0000 7f6f717fa640 1 -- 172.21.15.186:0/1231123255 <== mon.1 v2:172.21.15.186:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5110d1ad-5634-4c8e-9338-d6c4229b1d63", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f6f6c024350 con 0x7f6f741391d0 2024-05-04T17:44:22.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.962+0000 7f6f717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5110d1ad-5634-4c8e-9338-d6c4229b1d63", "id": 6}] 2024-05-04T17:44:22.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.962+0000 7f6f717fa640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:22.964 INFO:teuthology.orchestra.run.smithi186.stdout:6 2024-05-04T17:44:22.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 10 librados: watch_flush enter 2024-05-04T17:44:22.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 10 librados: watch_flush exit 2024-05-04T17:44:22.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 20 client.4156.objecter shutdown clearing up homeless session... 2024-05-04T17:44:22.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 10 client.4156.objecter successfully canceled tick 2024-05-04T17:44:22.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 10 monclient: shutdown 2024-05-04T17:44:22.965 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:22.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1231123255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f741391d0 msgr2=0x7f6f741c69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:22.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 1 --2- 172.21.15.186:0/1231123255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f741391d0 0x7f6f741c69f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00caa0 tx=0x7f6f6c00cf60 comp rx=0 tx=0).stop 2024-05-04T17:44:22.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1231123255 shutdown_connections 2024-05-04T17:44:22.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 1 --2- 172.21.15.186:0/1231123255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f741391d0 0x7f6f741c69f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:22.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 1 --2- 172.21.15.186:0/1231123255 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6f740b37a0 0x7f6f741d7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:22.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 1 --2- 172.21.15.186:0/1231123255 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6f740b2270 0x7f6f741d7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:22.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.963+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1231123255 >> 172.21.15.186:0/1231123255 conn(0x7f6f740b6950 msgr2=0x7f6f740b47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:22.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.964+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1231123255 shutdown_connections 2024-05-04T17:44:22.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.964+0000 7f6f7b0ba640 1 -- 172.21.15.186:0/1231123255 wait complete. 2024-05-04T17:44:22.966 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:22.964+0000 7f6f7b0ba640 1 librados: shutdown 2024-05-04T17:44:22.975 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new b307b905-5cc8-447b-8765-d888ed6f655b 7 2024-05-04T17:44:23.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.069+0000 7f14ecaac640 10 monclient: get_monmap_and_config 2024-05-04T17:44:23.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.069+0000 7f14ecaac640 10 monclient: build_initial_monmap 2024-05-04T17:44:23.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.069+0000 7f14ecaac640 10 monclient: monmap: 2024-05-04T17:44:23.070 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:23.070 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:23.070 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:44:23.070025+0000 2024-05-04T17:44:23.070 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:44:23.070025+0000 2024-05-04T17:44:23.070 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:23.070 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:23.070 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:23.070 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:23.071 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:23.071 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:23.071 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:23.071 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.069+0000 7f14ecaac640 1 Processor -- start 2024-05-04T17:44:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.069+0000 7f14ecaac640 1 -- start start 2024-05-04T17:44:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.069+0000 7f14ecaac640 10 monclient: init 2024-05-04T17:44:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-05-04T17:44:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f14e80a9720 0x7f14e80aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 10 monclient(hunting): picked mon.noname-c con 0x7f14e80a9720 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14e80a9b00 0x7f14e80b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 10 monclient(hunting): picked mon.noname-b con 0x7f14e80a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f14e8146890 0x7f14e813fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 10 monclient(hunting): picked mon.noname-a con 0x7f14e8146890 addr v2:172.21.15.19:3300/0 2024-05-04T17:44:23.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f14e80abc40 con 0x7f14e8146890 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f14e8146090 con 0x7f14e80a9720 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f14e8146370 con 0x7f14e80a9b00 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.070+0000 7f14ecaac640 10 monclient(hunting): authenticate will time out at 1259.348187s 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.071+0000 7f14e659c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f14e80a9720 0x7f14e80aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.071+0000 7f14e6d9d640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f14e8146890 0x7f14e813fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.071+0000 7f14e659c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f14e80a9720 0x7f14e80aa890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52812/0 (socket says 172.21.15.186:52812) 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.071+0000 7f14e659c640 1 -- 172.21.15.186:0/167994531 learned_addr learned my addr 172.21.15.186:0/167994531 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.071+0000 7f14e659c640 10 monclient(hunting): get_auth_request con 0x7f14e80a9720 auth_method 0 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.071+0000 7f14e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:23.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.071+0000 7f14e659c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:23.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.071+0000 7f14e659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:23.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.071+0000 7f14e6d9d640 10 monclient(hunting): get_auth_request con 0x7f14e8146890 auth_method 0 2024-05-04T17:44:23.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.071+0000 7f14e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:23.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.071+0000 7f14e6d9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:23.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.071+0000 7f14e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:23.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.072+0000 7f14e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:23.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.072+0000 7f14e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:23.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.072+0000 7f14e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:23.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.072+0000 7f14e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:23.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.072+0000 7f14e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:23.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.072+0000 7f14e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:23.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.073+0000 7f14e659c640 10 monclient(hunting): handle_auth_done global_id 4159 payload 274 2024-05-04T17:44:23.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.073+0000 7f14e659c640 1 -- 172.21.15.186:0/167994531 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14e80a9b00 msgr2=0x7f14e80b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:23.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.073+0000 7f14e659c640 1 --2- 172.21.15.186:0/167994531 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14e80a9b00 0x7f14e80b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.073+0000 7f14e659c640 1 -- 172.21.15.186:0/167994531 >> v2:172.21.15.19:3300/0 conn(0x7f14e8146890 msgr2=0x7f14e813fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:23.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.073+0000 7f14e659c640 1 --2- 172.21.15.186:0/167994531 >> v2:172.21.15.19:3300/0 conn(0x7f14e8146890 0x7f14e813fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.073+0000 7f14e659c640 10 monclient: _finish_hunting 0 2024-05-04T17:44:23.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.073+0000 7f14e659c640 1 monclient: found mon.noname-c 2024-05-04T17:44:23.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.073+0000 7f14e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:23.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.073+0000 7f14e659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:44:23.076 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.073+0000 7f14e659c640 1 -- 172.21.15.186:0/167994531 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e8146d20 con 0x7f14e80a9720 2024-05-04T17:44:23.077 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.075+0000 7f14e659c640 1 --2- 172.21.15.186:0/167994531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14e80a9720 0x7f14e80aa890 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f14e80b4bf0 tx=0x7f14d802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31e388925aa92852 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:23.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 1 -- 172.21.15.186:0/167994531 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d802ca70 con 0x7f14e80a9720 2024-05-04T17:44:23.081 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 10 monclient: dump: 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:23.082 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 10 monclient: _finish_auth 0 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 10 monclient: _check_auth_tickets 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 5 monclient: authenticate success, global_id 4159 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 1 -- 172.21.15.186:0/167994531 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d802cc10 con 0x7f14e80a9720 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 4 set_mon_vals no callback set 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 1 -- 172.21.15.186:0/167994531 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d80343f0 con 0x7f14e80a9720 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14e559a640 10 monclient: dump: 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:23.083 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 10 monclient: shutdown 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 1 -- 172.21.15.186:0/167994531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14e80a9720 msgr2=0x7f14e80aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 1 --2- 172.21.15.186:0/167994531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14e80a9720 0x7f14e80aa890 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f14e80b4bf0 tx=0x7f14d802bed0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 1 -- 172.21.15.186:0/167994531 shutdown_connections 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 1 --2- 172.21.15.186:0/167994531 >> v2:172.21.15.19:3300/0 conn(0x7f14e8146890 0x7f14e813fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 1 --2- 172.21.15.186:0/167994531 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14e80a9b00 0x7f14e80b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.084 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 1 --2- 172.21.15.186:0/167994531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14e80a9720 0x7f14e80aa890 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 1 -- 172.21.15.186:0/167994531 >> 172.21.15.186:0/167994531 conn(0x7f14e8067780 msgr2=0x7f14e80b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 1 -- 172.21.15.186:0/167994531 shutdown_connections 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.079+0000 7f14ecaac640 1 -- 172.21.15.186:0/167994531 wait complete. 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.080+0000 7f14ecaac640 10 monclient: build_initial_monmap 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.080+0000 7f14ecaac640 10 monclient: monmap: 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.080+0000 7f14ecaac640 1 librados: starting msgr at 2024-05-04T17:44:23.085 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.080+0000 7f14ecaac640 1 librados: starting objecter 2024-05-04T17:44:23.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 1 Processor -- start 2024-05-04T17:44:23.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 1 -- start start 2024-05-04T17:44:23.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 1 librados: setting wanted keys 2024-05-04T17:44:23.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 1 librados: calling monclient init 2024-05-04T17:44:23.086 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 10 monclient: init 2024-05-04T17:44:23.087 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14e80a9720 0x7f14e81cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 10 monclient(hunting): picked mon.noname-b con 0x7f14e80a9720 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14e80a9b00 0x7f14e81cc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 10 monclient(hunting): picked mon.noname-a con 0x7f14e80a9b00 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14e8146890 0x7f14e81cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 10 monclient(hunting): picked mon.noname-c con 0x7f14e8146890 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f14e8140770 con 0x7f14e80a9b00 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f14e81405f0 con 0x7f14e80a9720 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f14e8142b50 con 0x7f14e8146890 2024-05-04T17:44:23.088 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.081+0000 7f14ecaac640 10 monclient(hunting): authenticate will time out at 1259.359249s 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.082+0000 7f14e659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14e80a9720 0x7f14e81cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.082+0000 7f14e659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14e80a9720 0x7f14e81cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52816/0 (socket says 172.21.15.186:52816) 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.082+0000 7f14e659c640 1 -- 172.21.15.186:0/3079537182 learned_addr learned my addr 172.21.15.186:0/3079537182 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.082+0000 7f14e659c640 10 monclient(hunting): get_auth_request con 0x7f14e80a9720 auth_method 0 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.082+0000 7f14e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.082+0000 7f14e659c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.082+0000 7f14e659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.082+0000 7f14e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.082+0000 7f14e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.082+0000 7f14e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.085+0000 7f14e659c640 10 monclient(hunting): handle_auth_done global_id 4162 payload 824 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.085+0000 7f14e659c640 1 -- 172.21.15.186:0/3079537182 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14e8146890 msgr2=0x7f14e81cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.085+0000 7f14e659c640 1 --2- 172.21.15.186:0/3079537182 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14e8146890 0x7f14e81cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.085+0000 7f14e659c640 1 -- 172.21.15.186:0/3079537182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14e80a9b00 msgr2=0x7f14e81cc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.085+0000 7f14e659c640 1 --2- 172.21.15.186:0/3079537182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14e80a9b00 0x7f14e81cc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.089 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.085+0000 7f14e659c640 10 monclient: _finish_hunting 0 2024-05-04T17:44:23.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.085+0000 7f14e659c640 1 monclient: found mon.noname-b 2024-05-04T17:44:23.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.085+0000 7f14e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:23.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.085+0000 7f14e659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:23.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.085+0000 7f14e659c640 1 -- 172.21.15.186:0/3079537182 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e81ce740 con 0x7f14e80a9720 2024-05-04T17:44:23.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.087+0000 7f14e659c640 1 --2- 172.21.15.186:0/3079537182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14e80a9720 0x7f14e81cf1b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f14d8000f80 tx=0x7f14d8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:23.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.087+0000 7f14d6ffd640 10 client.?.objecter ms_handle_connect 0x7f14e80a9720 2024-05-04T17:44:23.090 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.087+0000 7f14d6ffd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:23.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 1 -- 172.21.15.186:0/3079537182 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d8043020 con 0x7f14e80a9720 2024-05-04T17:44:23.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:23.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:23.093 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 10 monclient: dump: 2024-05-04T17:44:23.093 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:23.093 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:23.093 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.093 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.093 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:23.093 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:23.093 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 10 monclient: _finish_auth 0 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 10 monclient: _check_auth_tickets 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 1 -- 172.21.15.186:0/3079537182 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d803e450 con 0x7f14e80a9720 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 1 -- 172.21.15.186:0/3079537182 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d8009510 con 0x7f14e80a9720 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14d6ffd640 10 monclient: dump: 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:23.094 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14ecaac640 5 monclient: authenticate success, global_id 4162 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14ecaac640 10 monclient: _renew_subs 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14ecaac640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14ecaac640 1 -- 172.21.15.186:0/3079537182 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14e81cfa40 con 0x7f14e80a9720 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14e4d99640 4 set_mon_vals no callback set 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14ecaac640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14ecaac640 10 monclient: _renew_subs 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14ecaac640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:23.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14ecaac640 1 -- 172.21.15.186:0/3079537182 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14e81ccfb0 con 0x7f14e80a9720 2024-05-04T17:44:23.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.091+0000 7f14ecaac640 1 librados: init done 2024-05-04T17:44:23.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.092+0000 7f14ecaac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:23.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.092+0000 7f14ecaac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:23.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.092+0000 7f14ecaac640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:23.096 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.092+0000 7f14ecaac640 1 -- 172.21.15.186:0/3079537182 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14e81cfc10 con 0x7f14e80a9720 2024-05-04T17:44:23.102 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.102+0000 7f14d6ffd640 1 -- 172.21.15.186:0/3079537182 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 2) ==== 45274+0+0 (secure 0 0 0) 0x7f14d8046020 con 0x7f14e80a9720 2024-05-04T17:44:23.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.103+0000 7f14d6ffd640 1 -- 172.21.15.186:0/3079537182 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f14d803dcd0 con 0x7f14e80a9720 2024-05-04T17:44:23.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.103+0000 7f14d6ffd640 10 client.4162.objecter ms_dispatch 0x7f14e8000b90 osd_map(9..9 src has 1..9) 2024-05-04T17:44:23.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.103+0000 7f14d6ffd640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-05-04T17:44:23.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.103+0000 7f14d6ffd640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-05-04T17:44:23.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.103+0000 7f14d6ffd640 20 client.4162.objecter dump_active .. 0 homeless 2024-05-04T17:44:23.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.204+0000 7f14d6ffd640 1 -- 172.21.15.186:0/3079537182 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f14e81cfc10 con 0x7f14e80a9720 2024-05-04T17:44:23.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.204+0000 7f14d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:23.205 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.204+0000 7f14d6ffd640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:23.297 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.297+0000 7f14ecaac640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b307b905-5cc8-447b-8765-d888ed6f655b", "id": 7}] 2024-05-04T17:44:23.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.297+0000 7f14ecaac640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b307b905-5cc8-447b-8765-d888ed6f655b", "id": 7}] 2024-05-04T17:44:23.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.297+0000 7f14ecaac640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:23.298 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.297+0000 7f14ecaac640 1 -- 172.21.15.186:0/3079537182 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b307b905-5cc8-447b-8765-d888ed6f655b", "id": 7} v 0) -- 0x7f14e806a700 con 0x7f14e80a9720 2024-05-04T17:44:23.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.655+0000 7f14d6ffd640 1 -- 172.21.15.186:0/3079537182 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b307b905-5cc8-447b-8765-d888ed6f655b", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f14d804d030 con 0x7f14e80a9720 2024-05-04T17:44:23.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.655+0000 7f14d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b307b905-5cc8-447b-8765-d888ed6f655b", "id": 7}] 2024-05-04T17:44:23.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.655+0000 7f14d6ffd640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:23.657 INFO:teuthology.orchestra.run.smithi186.stdout:7 2024-05-04T17:44:23.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 10 librados: watch_flush enter 2024-05-04T17:44:23.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 10 librados: watch_flush exit 2024-05-04T17:44:23.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 20 client.4162.objecter shutdown clearing up homeless session... 2024-05-04T17:44:23.658 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 10 client.4162.objecter successfully canceled tick 2024-05-04T17:44:23.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 10 monclient: shutdown 2024-05-04T17:44:23.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:23.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 1 -- 172.21.15.186:0/3079537182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14e80a9720 msgr2=0x7f14e81cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:23.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 1 --2- 172.21.15.186:0/3079537182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14e80a9720 0x7f14e81cf1b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f14d8000f80 tx=0x7f14d8004a40 comp rx=0 tx=0).stop 2024-05-04T17:44:23.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 1 -- 172.21.15.186:0/3079537182 shutdown_connections 2024-05-04T17:44:23.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 1 --2- 172.21.15.186:0/3079537182 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14e8146890 0x7f14e81cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 1 --2- 172.21.15.186:0/3079537182 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f14e80a9b00 0x7f14e81cc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 1 --2- 172.21.15.186:0/3079537182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14e80a9720 0x7f14e81cf1b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 1 -- 172.21.15.186:0/3079537182 >> 172.21.15.186:0/3079537182 conn(0x7f14e8067780 msgr2=0x7f14e80aa680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:23.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 1 -- 172.21.15.186:0/3079537182 shutdown_connections 2024-05-04T17:44:23.659 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 1 -- 172.21.15.186:0/3079537182 wait complete. 2024-05-04T17:44:23.660 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-04T17:44:23.656+0000 7f14ecaac640 1 librados: shutdown 2024-05-04T17:44:23.668 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-04T17:44:23.668 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-04T17:44:23.668 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-05-04T17:44:23.671 INFO:tasks.ceph.osd.0:Started 2024-05-04T17:44:23.671 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-04T17:44:23.671 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-04T17:44:23.671 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-05-04T17:44:23.673 INFO:tasks.ceph.osd.1:Started 2024-05-04T17:44:23.673 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-04T17:44:23.674 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-04T17:44:23.674 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-05-04T17:44:23.676 INFO:tasks.ceph.osd.2:Started 2024-05-04T17:44:23.676 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-04T17:44:23.676 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-04T17:44:23.676 DEBUG:teuthology.orchestra.run.smithi019:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-05-04T17:44:23.678 INFO:tasks.ceph.osd.3:Started 2024-05-04T17:44:23.679 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-04T17:44:23.679 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-04T17:44:23.679 DEBUG:teuthology.orchestra.run.smithi186:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-05-04T17:44:23.682 INFO:tasks.ceph.osd.4:Started 2024-05-04T17:44:23.682 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-04T17:44:23.682 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-04T17:44:23.682 DEBUG:teuthology.orchestra.run.smithi186:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-05-04T17:44:23.685 INFO:tasks.ceph.osd.5:Started 2024-05-04T17:44:23.685 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-04T17:44:23.685 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-04T17:44:23.685 DEBUG:teuthology.orchestra.run.smithi186:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-05-04T17:44:23.687 INFO:tasks.ceph.osd.6:Started 2024-05-04T17:44:23.687 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-05-04T17:44:23.687 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-04T17:44:23.687 DEBUG:teuthology.orchestra.run.smithi186:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-05-04T17:44:23.689 INFO:tasks.ceph.osd.7:Started 2024-05-04T17:44:23.689 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-04T17:44:23.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.851+0000 7fc0666c8640 10 monclient: get_monmap_and_config 2024-05-04T17:44:23.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.851+0000 7fc0666c8640 10 monclient: build_initial_monmap 2024-05-04T17:44:23.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.851+0000 7fc0666c8640 10 monclient: monmap: 2024-05-04T17:44:23.853 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:44:23.853 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:23.853 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:44:23.852461+0000 2024-05-04T17:44:23.853 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:44:23.852461+0000 2024-05-04T17:44:23.853 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:23.853 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:23.853 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:23.853 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:23.853 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:23.854 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:23.854 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:23.854 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:23.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.852+0000 7fc0666c8640 1 Processor -- start 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.852+0000 7fc0666c8640 1 -- start start 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.852+0000 7fc0666c8640 10 monclient: init 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.853+0000 7fc0666c8640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.853+0000 7fc0666c8640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.853+0000 7fc0666c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0600a9720 addr v1:172.21.15.19:6789/0 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.853+0000 7fc0666c8640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc0600a9b00 0x7fc0600aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.853+0000 7fc0666c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc0600a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.853+0000 7fc0666c8640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc0600b2e90 0x7fc06013fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.854+0000 7fc0666c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc0600b2e90 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.854+0000 7fc0666c8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0600abc40 con 0x7fc0600a9720 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.854+0000 7fc0666c8640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.854+0000 7fc0666c8640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fc060146fa0 con 0x7fc0600b2e90 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.854+0000 7fc0666c8640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.854+0000 7fc0666c8640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc060146090 con 0x7fc0600a9b00 2024-05-04T17:44:23.857 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.854+0000 7fc0666c8640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:23.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.854+0000 7fc0666c8640 10 monclient(hunting): authenticate will time out at 1285.968020s 2024-05-04T17:44:23.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.854+0000 7fc05ffff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fc0600a9720 0x7fc060146890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:38906/0 (socket says 172.21.15.19:38906) 2024-05-04T17:44:23.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.854+0000 7fc05ffff640 1 -- 172.21.15.19:0/2517663709 learned_addr learned my addr 172.21.15.19:0/2517663709 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:44:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.859+0000 7fc05effd640 1 -- 172.21.15.19:0/2517663709 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1349958926 0 0) 0x7fc0600abc40 con 0x7fc0600a9720 2024-05-04T17:44:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.859+0000 7fc05effd640 10 monclient(hunting): _negotiate 2024-05-04T17:44:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.859+0000 7fc05effd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.859+0000 7fc05effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.859+0000 7fc05effd640 10 monclient(hunting): my global_id is 4158 2024-05-04T17:44:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.859+0000 7fc05effd640 1 -- 172.21.15.19:0/2517663709 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc048003520 con 0x7fc0600a9720 2024-05-04T17:44:23.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.860+0000 7fc064c65640 1 --2- 172.21.15.19:0/2517663709 >> v2:172.21.15.186:3300/0 conn(0x7fc0600b2e90 0x7fc06013fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:23.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.862+0000 7fc064c65640 10 monclient(hunting): get_auth_request con 0x7fc0600b2e90 auth_method 0 2024-05-04T17:44:23.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.862+0000 7fc064c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:23.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.862+0000 7fc064c65640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:23.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.862+0000 7fc064c65640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:23.863 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.862+0000 7fc064c65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:23.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.862+0000 7fc064c65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:23.865 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.862+0000 7fc064c65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:23.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.865+0000 7fc064c65640 10 monclient(hunting): handle_auth_done global_id 4177 payload 274 2024-05-04T17:44:23.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.865+0000 7fc064c65640 1 -- 172.21.15.19:0/2517663709 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc0600a9b00 msgr2=0x7fc0600aa890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:23.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.865+0000 7fc064c65640 1 --2- 172.21.15.19:0/2517663709 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc0600a9b00 0x7fc0600aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.865+0000 7fc064c65640 1 -- 172.21.15.19:0/2517663709 >> v1:172.21.15.19:6789/0 conn(0x7fc0600a9720 legacy=0x7fc060146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:23.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.865+0000 7fc064c65640 10 monclient: _finish_hunting 0 2024-05-04T17:44:23.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.865+0000 7fc064c65640 1 monclient: found mon.noname-c 2024-05-04T17:44:23.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.865+0000 7fc064c65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:23.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.865+0000 7fc064c65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:44:23.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.865+0000 7fc064c65640 1 -- 172.21.15.19:0/2517663709 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc060146370 con 0x7fc0600b2e90 2024-05-04T17:44:23.869 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.868+0000 7fc064c65640 1 --2- 172.21.15.19:0/2517663709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0600b2e90 0x7fc06013fea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc058009d90 tx=0x7fc05802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc433e85f47206bf server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:23.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 1 -- 172.21.15.19:0/2517663709 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc058039070 con 0x7fc0600b2e90 2024-05-04T17:44:23.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:23.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:44:23.886 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 10 monclient: dump: 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 10 monclient: _finish_auth 0 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 10 monclient: _check_auth_tickets 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 1 -- 172.21.15.19:0/2517663709 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc05802cc60 con 0x7fc0600b2e90 2024-05-04T17:44:23.887 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc0666c8640 5 monclient: authenticate success, global_id 4177 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc0666c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 1 -- 172.21.15.19:0/2517663709 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc058034660 con 0x7fc0600b2e90 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc05effd640 10 monclient: dump: 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:23.888 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:23.889 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:23.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc0666c8640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:23.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.877+0000 7fc0666c8640 4 set_mon_vals no callback set 2024-05-04T17:44:23.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.878+0000 7fc0666c8640 10 monclient: shutdown 2024-05-04T17:44:23.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.878+0000 7fc0666c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:23.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.878+0000 7fc0666c8640 1 -- 172.21.15.19:0/2517663709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0600b2e90 msgr2=0x7fc06013fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:23.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.878+0000 7fc0666c8640 1 --2- 172.21.15.19:0/2517663709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0600b2e90 0x7fc06013fea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc058009d90 tx=0x7fc05802beb0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.878+0000 7fc0666c8640 1 -- 172.21.15.19:0/2517663709 shutdown_connections 2024-05-04T17:44:23.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.878+0000 7fc0666c8640 1 --2- 172.21.15.19:0/2517663709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0600b2e90 0x7fc06013fea0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.878+0000 7fc0666c8640 1 --2- 172.21.15.19:0/2517663709 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc0600a9b00 0x7fc0600aa890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.878+0000 7fc0666c8640 1 -- 172.21.15.19:0/2517663709 >> 172.21.15.19:0/2517663709 conn(0x7fc060067780 msgr2=0x7fc0600b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.878+0000 7fc0666c8640 1 -- 172.21.15.19:0/2517663709 shutdown_connections 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.878+0000 7fc0666c8640 1 -- 172.21.15.19:0/2517663709 wait complete. 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.879+0000 7fc0666c8640 10 monclient: build_initial_monmap 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.879+0000 7fc0666c8640 10 monclient: monmap: 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:23.890 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.880+0000 7fc0666c8640 1 librados: starting msgr at 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.880+0000 7fc0666c8640 1 librados: starting objecter 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.880+0000 7fc0666c8640 1 Processor -- start 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.880+0000 7fc0666c8640 1 -- start start 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.880+0000 7fc0666c8640 1 librados: setting wanted keys 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.880+0000 7fc0666c8640 1 librados: calling monclient init 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.880+0000 7fc0666c8640 10 monclient: init 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.882+0000 7fc0666c8640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.882+0000 7fc0666c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.882+0000 7fc0666c8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0600a9720 0x7fc060138180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.882+0000 7fc0666c8640 10 monclient(hunting): picked mon.noname-a con 0x7fc0600a9720 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.882+0000 7fc0666c8640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc0600a9b00 0x7fc0601db610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.882+0000 7fc0666c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc0600a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc0666c8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0600b2e90 0x7fc060135680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:23.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc0666c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc0600b2e90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc05ffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0600a9720 0x7fc060138180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc0666c8640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc0666c8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc060140770 con 0x7fc0600a9720 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc0666c8640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc0666c8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc0601405f0 con 0x7fc0600b2e90 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc0666c8640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc0666c8640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc06014d420 con 0x7fc0600a9b00 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc0666c8640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc0666c8640 10 monclient(hunting): authenticate will time out at 1285.997445s 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc05ffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0600a9720 0x7fc060138180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:32996/0 (socket says 172.21.15.19:32996) 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc05ffff640 1 -- 172.21.15.19:0/3663669026 learned_addr learned my addr 172.21.15.19:0/3663669026 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.883+0000 7fc064c65640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0600b2e90 0x7fc060135680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:23.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.884+0000 7fc05ffff640 10 monclient(hunting): get_auth_request con 0x7fc0600a9720 auth_method 0 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.884+0000 7fc05ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.884+0000 7fc05ffff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.884+0000 7fc05ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.884+0000 7fc064c65640 10 monclient(hunting): get_auth_request con 0x7fc0600b2e90 auth_method 0 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.884+0000 7fc064c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.884+0000 7fc064c65640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.884+0000 7fc064c65640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.885+0000 7fc064c65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.885+0000 7fc064c65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.885+0000 7fc064c65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.885+0000 7fc05ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.885+0000 7fc05ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:23.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.885+0000 7fc05ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:23.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.889+0000 7fc064c65640 10 monclient(hunting): handle_auth_done global_id 4180 payload 824 2024-05-04T17:44:23.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.889+0000 7fc064c65640 1 -- 172.21.15.19:0/3663669026 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc0600a9b00 msgr2=0x7fc0601db610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:23.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.889+0000 7fc064c65640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc0600a9b00 0x7fc0601db610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.889+0000 7fc064c65640 1 -- 172.21.15.19:0/3663669026 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0600a9720 msgr2=0x7fc060138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:23.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.889+0000 7fc064c65640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0600a9720 0x7fc060138180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:23.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.889+0000 7fc064c65640 10 monclient: _finish_hunting 0 2024-05-04T17:44:23.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.889+0000 7fc064c65640 1 monclient: found mon.noname-b 2024-05-04T17:44:23.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.889+0000 7fc064c65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:23.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.889+0000 7fc064c65640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:23.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.889+0000 7fc064c65640 1 -- 172.21.15.19:0/3663669026 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0601dc010 con 0x7fc0600b2e90 2024-05-04T17:44:23.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.893+0000 7fc064c65640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0600b2e90 0x7fc060135680 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc0600b91d0 tx=0x7fc058004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:23.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.893+0000 7fc05cff9640 10 client.?.objecter ms_handle_connect 0x7fc0600b2e90 2024-05-04T17:44:23.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.893+0000 7fc05cff9640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:23.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.899+0000 7fc05cff9640 1 -- 172.21.15.19:0/3663669026 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc058039050 con 0x7fc0600b2e90 2024-05-04T17:44:23.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.899+0000 7fc05cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.899+0000 7fc05cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.899+0000 7fc05cff9640 10 monclient: dump: 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.899+0000 7fc05cff9640 10 monclient: _finish_auth 0 2024-05-04T17:44:23.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.899+0000 7fc05cff9640 10 monclient: _check_auth_tickets 2024-05-04T17:44:23.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.899+0000 7fc05cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:23.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.899+0000 7fc05cff9640 1 -- 172.21.15.19:0/3663669026 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc05803a040 con 0x7fc0600b2e90 2024-05-04T17:44:23.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.899+0000 7fc05cff9640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:23.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.899+0000 7fc05e7fc640 4 set_mon_vals no callback set 2024-05-04T17:44:23.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc05cff9640 1 -- 172.21.15.19:0/3663669026 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc058009450 con 0x7fc0600b2e90 2024-05-04T17:44:23.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc0666c8640 5 monclient: authenticate success, global_id 4180 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc0666c8640 10 monclient: _renew_subs 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc0666c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc0666c8640 1 -- 172.21.15.19:0/3663669026 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc060141f30 con 0x7fc0600b2e90 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc05cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc05cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc0666c8640 10 client.4180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc05cff9640 10 monclient: dump: 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:23.903 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:23.904 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:23.904 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:23.904 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:44:23.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc0666c8640 10 monclient: _renew_subs 2024-05-04T17:44:23.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc0666c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:23.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc0666c8640 1 -- 172.21.15.19:0/3663669026 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc060135bc0 con 0x7fc0600b2e90 2024-05-04T17:44:23.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.900+0000 7fc0666c8640 1 librados: init done 2024-05-04T17:44:23.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.901+0000 7fc0666c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:23.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.901+0000 7fc0666c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:23.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.901+0000 7fc0666c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:23.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:23.901+0000 7fc0666c8640 1 -- 172.21.15.19:0/3663669026 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc06006a700 con 0x7fc0600b2e90 2024-05-04T17:44:24.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.019+0000 7fc05cff9640 1 -- 172.21.15.19:0/3663669026 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 2) ==== 45274+0+0 (secure 0 0 0) 0x7fc058047020 con 0x7fc0600b2e90 2024-05-04T17:44:24.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.071+0000 7fc05cff9640 1 -- 172.21.15.19:0/3663669026 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fc0580495d0 con 0x7fc0600b2e90 2024-05-04T17:44:24.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.071+0000 7fc05cff9640 10 client.4180.objecter ms_dispatch 0x7fc060000b90 osd_map(10..10 src has 1..10) 2024-05-04T17:44:24.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.071+0000 7fc05cff9640 3 client.4180.objecter handle_osd_map got epochs [10,10] > 0 2024-05-04T17:44:24.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.071+0000 7fc05cff9640 3 client.4180.objecter handle_osd_map decoding full epoch 10 2024-05-04T17:44:24.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.071+0000 7fc05cff9640 20 client.4180.objecter dump_active .. 0 homeless 2024-05-04T17:44:24.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.097+0000 7fc05cff9640 1 -- 172.21.15.19:0/3663669026 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc05803ed40 con 0x7fc0600b2e90 2024-05-04T17:44:24.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.097+0000 7fc05cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:24.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.097+0000 7fc05cff9640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.218+0000 7fc0666c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.218+0000 7fc0666c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.218+0000 7fc0666c8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.218+0000 7fc0666c8640 1 -- 172.21.15.19:0/3663669026 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0601422b0 con 0x7fc0600b2e90 2024-05-04T17:44:24.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.879+0000 7fc05cff9640 1 -- 172.21.15.19:0/3663669026 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mgrmap(e 3) ==== 90336+0+0 (secure 0 0 0) 0x7fc05803e800 con 0x7fc0600b2e90 2024-05-04T17:44:24.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.879+0000 7fc05cff9640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc04c06dc80 0x7fc04c070070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:24.880 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.880+0000 7fc05ffff640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc04c06dc80 0x7fc04c070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:24.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.880+0000 7fc05ffff640 10 monclient: get_auth_request con 0x7fc04c06dc80 auth_method 0 2024-05-04T17:44:24.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.880+0000 7fc05cff9640 10 client.4180.objecter ms_handle_connect 0x7fc04c06dc80 2024-05-04T17:44:24.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:24.880+0000 7fc05ffff640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc04c06dc80 0x7fc04c070070 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc050005880 tx=0x7fc050009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:25.032 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:25.031+0000 7f32e8d41640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-04T17:44:25.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.149+0000 7fc05cff9640 1 -- 172.21.15.19:0/3663669026 <== mon.1 v2:172.21.15.186:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fc058087640 con 0x7fc0600b2e90 2024-05-04T17:44:25.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.149+0000 7fc05cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:25.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.149+0000 7fc05cff9640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:25.150 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:44:25.150 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":10,"fsid":"18ced2df-a29f-46cc-b71a-1855bfb60c47","created":"2024-05-04T17:44:16.995866+0000","modified":"2024-05-04T17:44:23.464900+0000","last_up_change":"0.000000","last_in_change":"2024-05-04T17:44:23.464900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4b749ec0-22b6-458c-a7af-2ccace806ab7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3aa520fe-39c4-4fa2-860b-f4d0cc04ca97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16ed21c9-e5c3-4d65-b4d1-e4927a0fe919","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45aa2c0f-cfae-478c-a60c-4c1491e45525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb00f84f-d78c-461c-9843-73a9da4b80a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d07ad403-291f-4bde-a64a-5181fb79e544","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5110d1ad-5634-4c8e-9338-d6c4229b1d63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b307b905-5cc8-447b-8765-d888ed6f655b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:44:25.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 10 librados: watch_flush enter 2024-05-04T17:44:25.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 10 librados: watch_flush exit 2024-05-04T17:44:25.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 20 client.4180.objecter shutdown clearing up homeless session... 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 10 client.4180.objecter successfully canceled tick 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 1 -- 172.21.15.19:0/3663669026 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc04c06dc80 msgr2=0x7fc04c070070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc04c06dc80 0x7fc04c070070 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc050005880 tx=0x7fc050009040 comp rx=0 tx=0).stop 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 10 monclient: shutdown 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 1 -- 172.21.15.19:0/3663669026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0600b2e90 msgr2=0x7fc060135680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0600b2e90 0x7fc060135680 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc0600b91d0 tx=0x7fc058004a40 comp rx=0 tx=0).stop 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 1 -- 172.21.15.19:0/3663669026 shutdown_connections 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc04c06dc80 0x7fc04c070070 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0600b2e90 0x7fc060135680 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc0600a9b00 0x7fc0601db610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 1 --2- 172.21.15.19:0/3663669026 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc0600a9720 0x7fc060138180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:25.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 1 -- 172.21.15.19:0/3663669026 >> 172.21.15.19:0/3663669026 conn(0x7fc060067780 msgr2=0x7fc0600aa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:25.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 1 -- 172.21.15.19:0/3663669026 shutdown_connections 2024-05-04T17:44:25.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.150+0000 7fc0666c8640 1 -- 172.21.15.19:0/3663669026 wait complete. 2024-05-04T17:44:25.153 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:44:25.151+0000 7fc0666c8640 1 librados: shutdown 2024-05-04T17:44:25.166 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-04T17:44:25.166 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-04T17:44:25.216 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T17:44:25.216+0000 7f32e8d41640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-04T17:44:25.971 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-04T17:44:26.121 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.120+0000 7f182d73e640 10 monclient: get_monmap_and_config 2024-05-04T17:44:26.121 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.120+0000 7f182d73e640 10 monclient: build_initial_monmap 2024-05-04T17:44:26.121 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.120+0000 7f182d73e640 10 monclient: monmap: 2024-05-04T17:44:26.121 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:44:26.121 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:26.121 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:44:26.121209+0000 2024-05-04T17:44:26.121 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:44:26.121209+0000 2024-05-04T17:44:26.122 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:26.122 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:26.122 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:26.122 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:26.122 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:26.122 INFO:teuthology.misc.health.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:26.122 INFO:teuthology.misc.health.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:26.122 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:26.123 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.121+0000 7f182d73e640 1 Processor -- start 2024-05-04T17:44:26.123 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.121+0000 7f182d73e640 1 -- start start 2024-05-04T17:44:26.123 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.121+0000 7f182d73e640 10 monclient: init 2024-05-04T17:44:26.123 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.122+0000 7f182d73e640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:26.123 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.122+0000 7f182d73e640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-05-04T17:44:26.123 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.122+0000 7f182d73e640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f18280a96e0 0x7f182813fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:26.124 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.122+0000 7f182d73e640 10 monclient(hunting): picked mon.noname-a con 0x7f18280a96e0 addr v2:172.21.15.19:3300/0 2024-05-04T17:44:26.124 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.122+0000 7f182d73e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1828140260 addr v1:172.21.15.186:6789/0 2024-05-04T17:44:26.125 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.123+0000 7f182d73e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18281436c0 addr v1:172.21.15.19:6789/0 2024-05-04T17:44:26.125 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.123+0000 7f182d73e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18280ab940 con 0x7f18281436c0 2024-05-04T17:44:26.125 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.123+0000 7f182d73e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18280abbb0 con 0x7f1828140260 2024-05-04T17:44:26.125 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.123+0000 7f182d73e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:26.125 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.123+0000 7f182d73e640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f18280a9ac0 con 0x7f18280a96e0 2024-05-04T17:44:26.125 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.123+0000 7f182d73e640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:26.125 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.123+0000 7f182d73e640 10 monclient(hunting): authenticate will time out at 1288.236979s 2024-05-04T17:44:26.125 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.123+0000 7f1826ffd640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f18280a96e0 0x7f182813fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:26.126 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.123+0000 7f18267fc640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f1828140260 0x7f1828142fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:50078/0 (socket says 172.21.15.19:50078) 2024-05-04T17:44:26.126 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.123+0000 7f18267fc640 1 -- 172.21.15.19:0/3441809521 learned_addr learned my addr 172.21.15.19:0/3441809521 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:44:26.126 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.124+0000 7f1826ffd640 10 monclient(hunting): get_auth_request con 0x7f18280a96e0 auth_method 0 2024-05-04T17:44:26.126 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.124+0000 7f1826ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:26.126 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.124+0000 7f1826ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:26.126 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.124+0000 7f1826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:26.127 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.125+0000 7f1826ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:26.127 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.125+0000 7f1826ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:26.127 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.125+0000 7f1826ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:26.128 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.127+0000 7f1825ffb640 1 -- 172.21.15.19:0/3441809521 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3556041544 0 0) 0x7f18280abbb0 con 0x7f1828140260 2024-05-04T17:44:26.128 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.127+0000 7f1825ffb640 10 monclient(hunting): _negotiate 2024-05-04T17:44:26.128 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.127+0000 7f1825ffb640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:26.128 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.127+0000 7f1825ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:26.129 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.127+0000 7f1825ffb640 10 monclient(hunting): my global_id is 4183 2024-05-04T17:44:26.129 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.127+0000 7f1825ffb640 1 -- 172.21.15.19:0/3441809521 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1810003520 con 0x7f1828140260 2024-05-04T17:44:26.130 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.129+0000 7f1826ffd640 10 monclient(hunting): handle_auth_done global_id 4164 payload 274 2024-05-04T17:44:26.131 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.129+0000 7f1826ffd640 1 -- 172.21.15.19:0/3441809521 >> v1:172.21.15.186:6789/0 conn(0x7f1828140260 legacy=0x7f1828142fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:26.131 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.129+0000 7f1826ffd640 1 -- 172.21.15.19:0/3441809521 >> v1:172.21.15.19:6789/0 conn(0x7f18281436c0 legacy=0x7f182814d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:26.131 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.129+0000 7f1826ffd640 10 monclient: _finish_hunting 0 2024-05-04T17:44:26.131 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.129+0000 7f1826ffd640 1 monclient: found mon.noname-a 2024-05-04T17:44:26.131 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.129+0000 7f1826ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:26.131 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.129+0000 7f1826ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:44:26.131 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.129+0000 7f1826ffd640 1 -- 172.21.15.19:0/3441809521 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18280aa420 con 0x7f18280a96e0 2024-05-04T17:44:26.137 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.136+0000 7f1826ffd640 1 --2- 172.21.15.19:0/3441809521 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18280a96e0 0x7f182813fd20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f18180097a0 tx=0x7f181802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393e8ff38fbff4e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:26.150 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 1 -- 172.21.15.19:0/3441809521 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818038070 con 0x7f18280a96e0 2024-05-04T17:44:26.150 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:26.150 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:44:26.150 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 10 monclient: dump: 2024-05-04T17:44:26.150 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:26.150 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:26.150 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:26.150 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 10 monclient: _finish_auth 0 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 10 monclient: _check_auth_tickets 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f182d73e640 5 monclient: authenticate success, global_id 4164 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f182d73e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 1 -- 172.21.15.19:0/3441809521 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1818031700 con 0x7f18280a96e0 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 1 -- 172.21.15.19:0/3441809521 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818038070 con 0x7f18280a96e0 2024-05-04T17:44:26.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.146+0000 7f1825ffb640 10 monclient: dump: 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.147+0000 7f182d73e640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:26.152 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.147+0000 7f182d73e640 4 set_mon_vals no callback set 2024-05-04T17:44:26.153 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.148+0000 7f182d73e640 10 monclient: shutdown 2024-05-04T17:44:26.153 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.148+0000 7f182d73e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:26.153 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.148+0000 7f182d73e640 1 -- 172.21.15.19:0/3441809521 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18280a96e0 msgr2=0x7f182813fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:26.153 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.148+0000 7f182d73e640 1 --2- 172.21.15.19:0/3441809521 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18280a96e0 0x7f182813fd20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f18180097a0 tx=0x7f181802b7a0 comp rx=0 tx=0).stop 2024-05-04T17:44:26.159 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.149+0000 7f182d73e640 1 -- 172.21.15.19:0/3441809521 shutdown_connections 2024-05-04T17:44:26.159 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.149+0000 7f182d73e640 1 --2- 172.21.15.19:0/3441809521 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18280a96e0 0x7f182813fd20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:26.159 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.149+0000 7f182d73e640 1 -- 172.21.15.19:0/3441809521 >> 172.21.15.19:0/3441809521 conn(0x7f1828067960 msgr2=0x7f18280b4f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:26.159 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.149+0000 7f182d73e640 1 -- 172.21.15.19:0/3441809521 shutdown_connections 2024-05-04T17:44:26.159 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.150+0000 7f182d73e640 1 -- 172.21.15.19:0/3441809521 wait complete. 2024-05-04T17:44:26.159 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.151+0000 7f182d73e640 10 monclient: build_initial_monmap 2024-05-04T17:44:26.160 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.151+0000 7f182d73e640 10 monclient: monmap: 2024-05-04T17:44:26.160 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:44:26.160 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:26.160 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-05-04T17:44:26.160 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-05-04T17:44:26.160 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:26.160 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:26.160 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:26.160 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:26.160 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:26.160 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:26.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.152+0000 7f182d73e640 1 librados: starting msgr at 2024-05-04T17:44:26.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.152+0000 7f182d73e640 1 librados: starting objecter 2024-05-04T17:44:26.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.152+0000 7f182d73e640 1 Processor -- start 2024-05-04T17:44:26.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.152+0000 7f182d73e640 1 -- start start 2024-05-04T17:44:26.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.152+0000 7f182d73e640 1 librados: setting wanted keys 2024-05-04T17:44:26.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.153+0000 7f182d73e640 1 librados: calling monclient init 2024-05-04T17:44:26.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.153+0000 7f182d73e640 10 monclient: init 2024-05-04T17:44:26.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.154+0000 7f182d73e640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:26.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.154+0000 7f182d73e640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-04T17:44:26.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.154+0000 7f182d73e640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f18280a96e0 0x7f1828135660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:26.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.154+0000 7f182d73e640 10 monclient(hunting): picked mon.noname-c con 0x7f18280a96e0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:26.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.154+0000 7f182d73e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1828140260 0x7f18280b8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.154+0000 7f182d73e640 10 monclient(hunting): picked mon.noname-b con 0x7f1828140260 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.154+0000 7f182d73e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18281436c0 0x7f18281dba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.154+0000 7f182d73e640 10 monclient(hunting): picked mon.noname-a con 0x7f18281436c0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.154+0000 7f182d73e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.154+0000 7f182d73e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f18280aa870 con 0x7f18281436c0 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f182d73e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f182d73e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f18280b2e50 con 0x7f1828140260 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f182d73e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f1826ffd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f18280a96e0 0x7f1828135660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f18267fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1828140260 0x7f18280b8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f182d73e640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f182814eb90 con 0x7f18280a96e0 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f182d73e640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:26.162 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f182d73e640 10 monclient(hunting): authenticate will time out at 1288.269107s 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f18267fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1828140260 0x7f18280b8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:45108/0 (socket says 172.21.15.19:45108) 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f18267fc640 1 -- 172.21.15.19:0/3683059337 learned_addr learned my addr 172.21.15.19:0/3683059337 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f18267fc640 10 monclient(hunting): get_auth_request con 0x7f1828140260 auth_method 0 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f18267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f18267fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.155+0000 7f18267fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.156+0000 7f1826ffd640 10 monclient(hunting): get_auth_request con 0x7f18280a96e0 auth_method 0 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.156+0000 7f1826ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.156+0000 7f1826ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.156+0000 7f1826ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.156+0000 7f18277fe640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18281436c0 0x7f18281dba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.156+0000 7f18267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.156+0000 7f18267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.156+0000 7f18267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:26.163 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.156+0000 7f18277fe640 10 monclient(hunting): get_auth_request con 0x7f18281436c0 auth_method 0 2024-05-04T17:44:26.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.156+0000 7f18277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:26.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.156+0000 7f18277fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:26.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.156+0000 7f18277fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:26.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.157+0000 7f1826ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:26.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.157+0000 7f1826ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:26.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.157+0000 7f1826ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:26.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.157+0000 7f18277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:26.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.157+0000 7f18277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:26.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.157+0000 7f18277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:26.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.161+0000 7f18267fc640 10 monclient(hunting): handle_auth_done global_id 4186 payload 824 2024-05-04T17:44:26.166 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.161+0000 7f18267fc640 1 -- 172.21.15.19:0/3683059337 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f18280a96e0 msgr2=0x7f1828135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:26.166 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.161+0000 7f18267fc640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f18280a96e0 0x7f1828135660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:26.166 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.161+0000 7f18267fc640 1 -- 172.21.15.19:0/3683059337 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18281436c0 msgr2=0x7f18281dba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:26.166 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.161+0000 7f18267fc640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18281436c0 0x7f18281dba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:26.166 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.161+0000 7f18267fc640 10 monclient: _finish_hunting 0 2024-05-04T17:44:26.166 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.161+0000 7f18267fc640 1 monclient: found mon.noname-b 2024-05-04T17:44:26.166 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.161+0000 7f18267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:26.166 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.161+0000 7f18267fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:26.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.161+0000 7f18267fc640 1 -- 172.21.15.19:0/3683059337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18281369a0 con 0x7f1828140260 2024-05-04T17:44:26.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.164+0000 7f18267fc640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1828140260 0x7f18280b8e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f181c00c460 tx=0x7f181c00c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:26.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.165+0000 7f180f7fe640 10 client.?.objecter ms_handle_connect 0x7f1828140260 2024-05-04T17:44:26.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.165+0000 7f180f7fe640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:26.174 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 1 -- 172.21.15.19:0/3683059337 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c017070 con 0x7f1828140260 2024-05-04T17:44:26.174 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:26.174 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:26.174 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 10 monclient: dump: 2024-05-04T17:44:26.174 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:26.175 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:26.175 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:26.175 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:26.175 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:26.175 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:26.175 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:26.175 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:26.176 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:26.176 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:26.176 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 10 monclient: _finish_auth 0 2024-05-04T17:44:26.176 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 10 monclient: _check_auth_tickets 2024-05-04T17:44:26.176 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:26.176 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 1 -- 172.21.15.19:0/3683059337 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f181c005ce0 con 0x7f1828140260 2024-05-04T17:44:26.177 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:26.177 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 1 -- 172.21.15.19:0/3683059337 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c007290 con 0x7f1828140260 2024-05-04T17:44:26.177 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:26.177 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:26.177 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f180f7fe640 10 monclient: dump: 2024-05-04T17:44:26.177 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.170+0000 7f18257fa640 4 set_mon_vals no callback set 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.171+0000 7f182d73e640 5 monclient: authenticate success, global_id 4186 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.171+0000 7f182d73e640 10 monclient: _renew_subs 2024-05-04T17:44:26.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.171+0000 7f182d73e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:26.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.171+0000 7f182d73e640 1 -- 172.21.15.19:0/3683059337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1828136c80 con 0x7f1828140260 2024-05-04T17:44:26.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.171+0000 7f182d73e640 10 client.4186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:26.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.171+0000 7f182d73e640 10 monclient: _renew_subs 2024-05-04T17:44:26.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.171+0000 7f182d73e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:26.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.171+0000 7f182d73e640 1 -- 172.21.15.19:0/3683059337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18280b6000 con 0x7f1828140260 2024-05-04T17:44:26.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.171+0000 7f182d73e640 1 librados: init done 2024-05-04T17:44:26.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.172+0000 7f182d73e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:26.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.172+0000 7f182d73e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:26.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.172+0000 7f182d73e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:26.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.172+0000 7f182d73e640 1 -- 172.21.15.19:0/3683059337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f182804f5f0 con 0x7f1828140260 2024-05-04T17:44:26.624 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.623+0000 7f180f7fe640 1 -- 172.21.15.19:0/3683059337 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f181c01a070 con 0x7f1828140260 2024-05-04T17:44:26.624 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.623+0000 7f180f7fe640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f181406d730 0x7f181406fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:26.624 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.624+0000 7f1826ffd640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f181406d730 0x7f181406fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:26.625 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.624+0000 7f1826ffd640 10 monclient: get_auth_request con 0x7f181406d730 auth_method 0 2024-05-04T17:44:26.625 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.625+0000 7f180f7fe640 10 client.4186.objecter ms_handle_connect 0x7f181406d730 2024-05-04T17:44:26.625 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.625+0000 7f1826ffd640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f181406d730 0x7f181406fbf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1818035000 tx=0x7f1818008030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:26.628 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.628+0000 7f180f7fe640 1 -- 172.21.15.19:0/3683059337 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f181c0575a0 con 0x7f1828140260 2024-05-04T17:44:26.629 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.628+0000 7f180f7fe640 10 client.4186.objecter ms_dispatch 0x7f1828000b90 osd_map(10..10 src has 1..10) 2024-05-04T17:44:26.629 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.628+0000 7f180f7fe640 3 client.4186.objecter handle_osd_map got epochs [10,10] > 0 2024-05-04T17:44:26.629 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.628+0000 7f180f7fe640 3 client.4186.objecter handle_osd_map decoding full epoch 10 2024-05-04T17:44:26.629 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.628+0000 7f180f7fe640 20 client.4186.objecter dump_active .. 0 homeless 2024-05-04T17:44:26.649 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.648+0000 7f180f7fe640 1 -- 172.21.15.19:0/3683059337 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f181c021020 con 0x7f1828140260 2024-05-04T17:44:26.649 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.648+0000 7f180f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:26.649 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:26.648+0000 7f180f7fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:27.024 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.023+0000 7f182d73e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:27.024 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.023+0000 7f182d73e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:27.024 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.023+0000 7f182d73e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:27.024 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.023+0000 7f182d73e640 1 -- 172.21.15.19:0/3683059337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f182813edf0 con 0x7f1828140260 2024-05-04T17:44:27.053 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.052+0000 7f180f7fe640 1 -- 172.21.15.19:0/3683059337 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f181c058910 con 0x7f1828140260 2024-05-04T17:44:27.053 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.052+0000 7f180f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:27.053 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.052+0000 7f180f7fe640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:27.053 INFO:teuthology.misc.health.smithi019.stdout: 2024-05-04T17:44:27.054 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"18ced2df-a29f-46cc-b71a-1855bfb60c47","created":"2024-05-04T17:44:16.995866+0000","modified":"2024-05-04T17:44:23.464900+0000","last_up_change":"0.000000","last_in_change":"2024-05-04T17:44:23.464900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4b749ec0-22b6-458c-a7af-2ccace806ab7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3aa520fe-39c4-4fa2-860b-f4d0cc04ca97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16ed21c9-e5c3-4d65-b4d1-e4927a0fe919","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45aa2c0f-cfae-478c-a60c-4c1491e45525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb00f84f-d78c-461c-9843-73a9da4b80a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d07ad403-291f-4bde-a64a-5181fb79e544","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5110d1ad-5634-4c8e-9338-d6c4229b1d63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b307b905-5cc8-447b-8765-d888ed6f655b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:44:27.057 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 10 librados: watch_flush enter 2024-05-04T17:44:27.057 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 10 librados: watch_flush exit 2024-05-04T17:44:27.057 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 20 client.4186.objecter shutdown clearing up homeless session... 2024-05-04T17:44:27.058 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 10 client.4186.objecter successfully canceled tick 2024-05-04T17:44:27.058 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 1 -- 172.21.15.19:0/3683059337 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f181406d730 msgr2=0x7f181406fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:27.058 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f181406d730 0x7f181406fbf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1818035000 tx=0x7f1818008030 comp rx=0 tx=0).stop 2024-05-04T17:44:27.058 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 10 monclient: shutdown 2024-05-04T17:44:27.058 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:27.058 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 1 -- 172.21.15.19:0/3683059337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1828140260 msgr2=0x7f18280b8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:27.058 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1828140260 0x7f18280b8e00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f181c00c460 tx=0x7f181c00c920 comp rx=0 tx=0).stop 2024-05-04T17:44:27.058 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 1 -- 172.21.15.19:0/3683059337 shutdown_connections 2024-05-04T17:44:27.058 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f181406d730 0x7f181406fbf0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:27.058 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f18281436c0 0x7f18281dba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:27.058 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1828140260 0x7f18280b8e00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:27.059 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 1 --2- 172.21.15.19:0/3683059337 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f18280a96e0 0x7f1828135660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:27.059 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 1 -- 172.21.15.19:0/3683059337 >> 172.21.15.19:0/3683059337 conn(0x7f1828067960 msgr2=0x7f18280b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:27.059 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 1 -- 172.21.15.19:0/3683059337 shutdown_connections 2024-05-04T17:44:27.059 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.055+0000 7f182d73e640 1 -- 172.21.15.19:0/3683059337 wait complete. 2024-05-04T17:44:27.059 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:27.056+0000 7f182d73e640 1 librados: shutdown 2024-05-04T17:44:27.070 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-04T17:44:30.980 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:07.232 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.060 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:07.312 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.169 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:07.426 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.228 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:07.486 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.229 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:07.485 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.287 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:07.546 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.342 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:07.572 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.401 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:07.631 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.462 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:07.717 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.488 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:07.728 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.526 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:07.781 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.549 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:07.789 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.570 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:07.806 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:44:31.585+0000 5dce500 -1 Falling back to public interface 2024-05-04T17:44:31.621 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:07.862 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.630 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:07.866 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.682 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:07.923 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:31.812 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-05-04T17:44:31.804+0000 5dce500 -1 Falling back to public interface 2024-05-04T17:44:31.838 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:44:31.831+0000 5dce500 -1 Falling back to public interface 2024-05-04T17:44:31.970 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:44:31.964+0000 5dce500 -1 Falling back to public interface 2024-05-04T17:44:32.043 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:08.295 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.077 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:44:32.069+0000 5dce500 -1 Falling back to public interface 2024-05-04T17:44:32.109 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:44:32.099+0000 5dce500 -1 Falling back to public interface 2024-05-04T17:44:32.231 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:44:32.221+0000 5dce500 -1 Falling back to public interface 2024-05-04T17:44:32.253 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:44:32.243+0000 5dce500 -1 Falling back to public interface 2024-05-04T17:44:32.303 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:08.559 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.319 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:08.577 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.435 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:08.688 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.460 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:08.690 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.534 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:08.789 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.605 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:08.845 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.670 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:08.926 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.688 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:08.946 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.732 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:08.968 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.751 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:08.992 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.776 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:09.028 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.831 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:09.061 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.905 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:09.160 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.979 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:09.219 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:32.991 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:09.247 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.062 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:09.320 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.074 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:09.326 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.107 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:09.343 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.128 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:09.369 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.145 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:09.375 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.258 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:09.514 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.269 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:09.524 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.338 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:09.577 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.350 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:09.608 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.369 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:09.621 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.414 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:09.644 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.426 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:09.662 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.450 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:09.691 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.522 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:09.778 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.558 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:09.813 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.626 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:09.866 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.638 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:09.896 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.660 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:09.912 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.686 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:09.916 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.694 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:09.930 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.724 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:09.965 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.786 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:10.042 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.851 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:10.106 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.876 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-04T17:44:33.926 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:10.184 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.932 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:10.172 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.952 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:10.204 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.959 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:10.189 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.966 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:10.202 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:33.981 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.979+0000 7f149d40f640 10 monclient: get_monmap_and_config 2024-05-04T17:44:33.981 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.979+0000 7f149d40f640 10 monclient: build_initial_monmap 2024-05-04T17:44:33.981 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.979+0000 7f149d40f640 10 monclient: monmap: 2024-05-04T17:44:33.981 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:44:33.981 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:33.981 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:44:33.981567+0000 2024-05-04T17:44:33.982 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:44:33.981567+0000 2024-05-04T17:44:33.982 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:33.982 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:33.982 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:33.982 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:33.982 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:33.982 INFO:teuthology.misc.health.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:33.982 INFO:teuthology.misc.health.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:33.982 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:33.984 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.980+0000 7f149d40f640 1 Processor -- start 2024-05-04T17:44:33.984 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.980+0000 7f149d40f640 1 -- start start 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.980+0000 7f149d40f640 10 monclient: init 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.980+0000 7f149d40f640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.980+0000 7f149d40f640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.980+0000 7f149d40f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f14980a96e0 0x7f14981430d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.980+0000 7f149d40f640 10 monclient(hunting): picked mon.noname-c con 0x7f14980a96e0 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.980+0000 7f149d40f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14980a9ac0 addr v1:172.21.15.19:6789/0 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f149d40f640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1498143610 0x7f1498146840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f149d40f640 10 monclient(hunting): picked mon.noname-b con 0x7f1498143610 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f149d40f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14980ab910 con 0x7f14980a9ac0 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f149d40f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f149d40f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f14980abbf0 con 0x7f14980a96e0 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f149d40f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f149d40f640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f1498142bb0 con 0x7f1498143610 2024-05-04T17:44:33.985 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f149d40f640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f149d40f640 10 monclient(hunting): authenticate will time out at 1296.095668s 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f14977fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1498143610 0x7f1498146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f1496ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f14980a96e0 0x7f14981430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f14967fc640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f14980a9ac0 0x7f1498146130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:48530/0 (socket says 172.21.15.19:48530) 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f14967fc640 1 -- 172.21.15.19:0/2777166227 learned_addr learned my addr 172.21.15.19:0/2777166227 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f14977fe640 10 monclient(hunting): get_auth_request con 0x7f1498143610 auth_method 0 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f14977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f14977fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.981+0000 7f14977fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.982+0000 7f1496ffd640 10 monclient(hunting): get_auth_request con 0x7f14980a96e0 auth_method 0 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.982+0000 7f1496ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.982+0000 7f1496ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.982+0000 7f1496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:33.986 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.982+0000 7f14977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:33.987 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.982+0000 7f14977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:33.987 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.982+0000 7f14977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:33.987 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.982+0000 7f1496ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:33.987 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.982+0000 7f1496ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:33.987 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.982+0000 7f1496ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:33.987 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.984+0000 7f1496ffd640 10 monclient(hunting): handle_auth_done global_id 4219 payload 274 2024-05-04T17:44:33.988 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.984+0000 7f1496ffd640 1 -- 172.21.15.19:0/2777166227 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1498143610 msgr2=0x7f1498146840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:33.988 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.984+0000 7f1496ffd640 1 --2- 172.21.15.19:0/2777166227 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1498143610 0x7f1498146840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:33.988 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.984+0000 7f1496ffd640 1 -- 172.21.15.19:0/2777166227 >> v1:172.21.15.19:6789/0 conn(0x7f14980a9ac0 legacy=0x7f1498146130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:33.988 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.984+0000 7f1496ffd640 10 monclient: _finish_hunting 0 2024-05-04T17:44:33.988 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.984+0000 7f1496ffd640 1 monclient: found mon.noname-c 2024-05-04T17:44:33.988 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.984+0000 7f1496ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:33.988 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.984+0000 7f1496ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:44:33.988 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.984+0000 7f1496ffd640 1 -- 172.21.15.19:0/2777166227 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1498143aa0 con 0x7f14980a96e0 2024-05-04T17:44:33.988 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.985+0000 7f1496ffd640 1 --2- 172.21.15.19:0/2777166227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980a96e0 0x7f14981430d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f14880097a0 tx=0x7f148802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d152738ab2f8acbc server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:33.991 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 1 -- 172.21.15.19:0/2777166227 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488035020 con 0x7f14980a96e0 2024-05-04T17:44:33.991 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:33.991 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:44:33.991 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 10 monclient: dump: 2024-05-04T17:44:33.991 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:33.991 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:33.991 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:33.991 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:33.991 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:33.991 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 10 monclient: _finish_auth 0 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 10 monclient: _check_auth_tickets 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 1 -- 172.21.15.19:0/2777166227 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1488031830 con 0x7f14980a96e0 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 1 -- 172.21.15.19:0/2777166227 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488031c00 con 0x7f14980a96e0 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:33.992 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f1495ffb640 10 monclient: dump: 2024-05-04T17:44:33.993 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:33.993 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:33.993 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:33.993 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:33.993 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:33.993 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:33.993 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:33.993 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:33.993 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:33.993 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:33.993 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f149d40f640 5 monclient: authenticate success, global_id 4219 2024-05-04T17:44:33.993 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f149d40f640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:33.994 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.989+0000 7f149d40f640 4 set_mon_vals no callback set 2024-05-04T17:44:33.997 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.990+0000 7f149d40f640 10 monclient: shutdown 2024-05-04T17:44:33.997 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.990+0000 7f149d40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:33.997 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.990+0000 7f149d40f640 1 -- 172.21.15.19:0/2777166227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980a96e0 msgr2=0x7f14981430d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:33.997 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.990+0000 7f149d40f640 1 --2- 172.21.15.19:0/2777166227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980a96e0 0x7f14981430d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f14880097a0 tx=0x7f148802b8d0 comp rx=0 tx=0).stop 2024-05-04T17:44:33.997 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.990+0000 7f149d40f640 1 -- 172.21.15.19:0/2777166227 shutdown_connections 2024-05-04T17:44:33.997 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.990+0000 7f149d40f640 1 --2- 172.21.15.19:0/2777166227 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1498143610 0x7f1498146840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.990+0000 7f149d40f640 1 --2- 172.21.15.19:0/2777166227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980a96e0 0x7f14981430d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.990+0000 7f149d40f640 1 -- 172.21.15.19:0/2777166227 >> 172.21.15.19:0/2777166227 conn(0x7f14980678d0 msgr2=0x7f14980b4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.990+0000 7f149d40f640 1 -- 172.21.15.19:0/2777166227 shutdown_connections 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.990+0000 7f149d40f640 1 -- 172.21.15.19:0/2777166227 wait complete. 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.990+0000 7f149d40f640 10 monclient: build_initial_monmap 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.990+0000 7f149d40f640 10 monclient: monmap: 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:33.998 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 1 librados: starting msgr at 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 1 librados: starting objecter 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 1 Processor -- start 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 1 -- start start 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 1 librados: setting wanted keys 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 1 librados: calling monclient init 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 10 monclient: init 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980a96e0 0x7f14980b7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 10 monclient(hunting): picked mon.noname-b con 0x7f14980a96e0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14980a9ac0 0x7f1498145520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:33.999 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 10 monclient(hunting): picked mon.noname-c con 0x7f14980a9ac0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1498143610 0x7f1498145a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 10 monclient(hunting): picked mon.noname-a con 0x7f1498143610 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f14980aaaa0 con 0x7f1498143610 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f14980aa920 con 0x7f14980a96e0 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f1498146d80 con 0x7f14980a9ac0 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.991+0000 7f149d40f640 10 monclient(hunting): authenticate will time out at 1296.106599s 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f14967fc640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14980a9ac0 0x7f1498145520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f1496ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980a96e0 0x7f14980b7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f14977fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1498143610 0x7f1498145a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f14967fc640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14980a9ac0 0x7f1498145520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:46550/0 (socket says 172.21.15.19:46550) 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f14967fc640 1 -- 172.21.15.19:0/350094121 learned_addr learned my addr 172.21.15.19:0/350094121 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:44:34.000 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f14967fc640 10 monclient(hunting): get_auth_request con 0x7f14980a9ac0 auth_method 0 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f14967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f14967fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f14967fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f1496ffd640 10 monclient(hunting): get_auth_request con 0x7f14980a96e0 auth_method 0 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f1496ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f1496ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f1496ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f14977fe640 10 monclient(hunting): get_auth_request con 0x7f1498143610 auth_method 0 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f14977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f14977fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.992+0000 7f14977fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.993+0000 7f1496ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.993+0000 7f1496ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.993+0000 7f1496ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:34.001 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.993+0000 7f14967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.993+0000 7f14967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.993+0000 7f14967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.993+0000 7f14977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.993+0000 7f14977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.993+0000 7f14977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.995+0000 7f1496ffd640 10 monclient(hunting): handle_auth_done global_id 4222 payload 824 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.995+0000 7f1496ffd640 1 -- 172.21.15.19:0/350094121 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14980a9ac0 msgr2=0x7f1498145520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.995+0000 7f1496ffd640 1 --2- 172.21.15.19:0/350094121 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14980a9ac0 0x7f1498145520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.995+0000 7f1496ffd640 1 -- 172.21.15.19:0/350094121 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1498143610 msgr2=0x7f1498145a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.995+0000 7f1496ffd640 1 --2- 172.21.15.19:0/350094121 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1498143610 0x7f1498145a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.995+0000 7f1496ffd640 10 monclient: _finish_hunting 0 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.995+0000 7f1496ffd640 1 monclient: found mon.noname-b 2024-05-04T17:44:34.002 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.995+0000 7f1496ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:34.003 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.995+0000 7f1496ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:34.003 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.995+0000 7f1496ffd640 1 -- 172.21.15.19:0/350094121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14980b73d0 con 0x7f14980a96e0 2024-05-04T17:44:34.003 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:10.239 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.004 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.997+0000 7f1496ffd640 1 --2- 172.21.15.19:0/350094121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980a96e0 0x7f14980b7b90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1488033000 tx=0x7f14880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.997+0000 7f147f7fe640 10 client.?.objecter ms_handle_connect 0x7f14980a96e0 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:33.997+0000 7f147f7fe640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 1 -- 172.21.15.19:0/350094121 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488035040 con 0x7f14980a96e0 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 10 monclient: dump: 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:34.005 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 10 monclient: _finish_auth 0 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 10 monclient: _check_auth_tickets 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f149d40f640 5 monclient: authenticate success, global_id 4222 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 1 -- 172.21.15.19:0/350094121 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f148803b920 con 0x7f14980a96e0 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f149d40f640 10 monclient: _renew_subs 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f149d40f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f149d40f640 1 -- 172.21.15.19:0/350094121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14980b76d0 con 0x7f14980a96e0 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 1 -- 172.21.15.19:0/350094121 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148803bd10 con 0x7f14980a96e0 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:34.006 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f147f7fe640 10 monclient: dump: 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f149d40f640 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f149d40f640 10 monclient: _renew_subs 2024-05-04T17:44:34.007 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f149d40f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:34.008 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f149d40f640 1 -- 172.21.15.19:0/350094121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1498143f30 con 0x7f14980a96e0 2024-05-04T17:44:34.008 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f149d40f640 1 librados: init done 2024-05-04T17:44:34.008 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.001+0000 7f14957fa640 4 set_mon_vals no callback set 2024-05-04T17:44:34.008 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.002+0000 7f149d40f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:34.008 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.002+0000 7f149d40f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:34.008 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.002+0000 7f149d40f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:34.008 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.002+0000 7f149d40f640 1 -- 172.21.15.19:0/350094121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f149804f5f0 con 0x7f14980a96e0 2024-05-04T17:44:34.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.021+0000 7f147f7fe640 1 -- 172.21.15.19:0/350094121 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f1488038020 con 0x7f14980a96e0 2024-05-04T17:44:34.023 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.021+0000 7f147f7fe640 1 --2- 172.21.15.19:0/350094121 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f148406d730 0x7f148406fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:34.023 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.021+0000 7f14967fc640 1 --2- 172.21.15.19:0/350094121 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f148406d730 0x7f148406fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:34.023 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.021+0000 7f14967fc640 10 monclient: get_auth_request con 0x7f148406d730 auth_method 0 2024-05-04T17:44:34.023 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.022+0000 7f147f7fe640 10 client.4222.objecter ms_handle_connect 0x7f148406d730 2024-05-04T17:44:34.023 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.022+0000 7f14967fc640 1 --2- 172.21.15.19:0/350094121 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f148406d730 0x7f148406fbf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f149813ed10 tx=0x7f148c007f10 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:34.025 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.023+0000 7f147f7fe640 1 -- 172.21.15.19:0/350094121 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f14880ab9b0 con 0x7f14980a96e0 2024-05-04T17:44:34.025 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.023+0000 7f147f7fe640 10 client.4222.objecter ms_dispatch 0x7f1498000b90 osd_map(10..10 src has 1..10) 2024-05-04T17:44:34.025 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.023+0000 7f147f7fe640 3 client.4222.objecter handle_osd_map got epochs [10,10] > 0 2024-05-04T17:44:34.025 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.023+0000 7f147f7fe640 3 client.4222.objecter handle_osd_map decoding full epoch 10 2024-05-04T17:44:34.025 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.023+0000 7f147f7fe640 20 client.4222.objecter dump_active .. 0 homeless 2024-05-04T17:44:34.051 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:10.307 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.149 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:10.404 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.214 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:10.472 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.238 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:10.474 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.241 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:10.471 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.243 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:10.483 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.259 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:10.511 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.268 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:10.509 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.290 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.288+0000 7f147f7fe640 1 -- 172.21.15.19:0/350094121 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1488054020 con 0x7f14980a96e0 2024-05-04T17:44:34.290 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.288+0000 7f147f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:34.290 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.288+0000 7f147f7fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:34.318 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:10.571 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.439 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:10.694 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.489 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.487+0000 7f149d40f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:34.489 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.487+0000 7f149d40f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:34.489 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.487+0000 7f149d40f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:34.489 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.487+0000 7f149d40f640 1 -- 172.21.15.19:0/350094121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14980b78b0 con 0x7f14980a96e0 2024-05-04T17:44:34.505 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.503+0000 7f147f7fe640 1 -- 172.21.15.19:0/350094121 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f14980b76d0 con 0x7f14980a96e0 2024-05-04T17:44:34.506 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.504+0000 7f147f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:34.506 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.504+0000 7f147f7fe640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:34.506 INFO:teuthology.misc.health.smithi019.stdout: 2024-05-04T17:44:34.506 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"18ced2df-a29f-46cc-b71a-1855bfb60c47","created":"2024-05-04T17:44:16.995866+0000","modified":"2024-05-04T17:44:23.464900+0000","last_up_change":"0.000000","last_in_change":"2024-05-04T17:44:23.464900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4b749ec0-22b6-458c-a7af-2ccace806ab7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3aa520fe-39c4-4fa2-860b-f4d0cc04ca97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16ed21c9-e5c3-4d65-b4d1-e4927a0fe919","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45aa2c0f-cfae-478c-a60c-4c1491e45525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb00f84f-d78c-461c-9843-73a9da4b80a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d07ad403-291f-4bde-a64a-5181fb79e544","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5110d1ad-5634-4c8e-9338-d6c4229b1d63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b307b905-5cc8-447b-8765-d888ed6f655b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:44:34.508 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 10 librados: watch_flush enter 2024-05-04T17:44:34.508 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 10 librados: watch_flush exit 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 20 client.4222.objecter shutdown clearing up homeless session... 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 10 client.4222.objecter successfully canceled tick 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 -- 172.21.15.19:0/350094121 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f148406d730 msgr2=0x7f148406fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 --2- 172.21.15.19:0/350094121 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f148406d730 0x7f148406fbf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f149813ed10 tx=0x7f148c007f10 comp rx=0 tx=0).stop 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 10 monclient: shutdown 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 -- 172.21.15.19:0/350094121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980a96e0 msgr2=0x7f14980b7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 --2- 172.21.15.19:0/350094121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980a96e0 0x7f14980b7b90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1488033000 tx=0x7f14880046c0 comp rx=0 tx=0).stop 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 -- 172.21.15.19:0/350094121 shutdown_connections 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 --2- 172.21.15.19:0/350094121 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f148406d730 0x7f148406fbf0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 --2- 172.21.15.19:0/350094121 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1498143610 0x7f1498145a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 --2- 172.21.15.19:0/350094121 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f14980a9ac0 0x7f1498145520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 --2- 172.21.15.19:0/350094121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980a96e0 0x7f14980b7b90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:34.509 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 -- 172.21.15.19:0/350094121 >> 172.21.15.19:0/350094121 conn(0x7f14980678d0 msgr2=0x7f149813de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:34.510 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 -- 172.21.15.19:0/350094121 shutdown_connections 2024-05-04T17:44:34.510 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 -- 172.21.15.19:0/350094121 wait complete. 2024-05-04T17:44:34.510 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:34.506+0000 7f149d40f640 1 librados: shutdown 2024-05-04T17:44:34.511 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:10.741 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.517 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-04T17:44:34.518 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:10.753 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.542 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:10.783 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.550 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:10.790 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.553 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:10.812 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.557 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:10.810 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.588 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:10.844 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.729 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:10.983 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.783 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:11.013 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.784 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:11.019 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.807 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:11.048 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.843 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:11.083 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:34.848 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:11.106 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.027 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:11.282 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.053 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:11.289 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.078 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:11.334 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.078 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:11.331 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.079 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:11.320 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.133 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:11.385 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.138 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:11.377 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.158 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:11.414 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.281 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:11.511 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.339 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:11.569 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.422 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:11.681 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.477 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:11.735 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.543 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:11.779 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.577 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:11.832 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.602 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:11.843 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.614 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:11.850 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.637 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:11.892 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.658 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:11.900 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.673 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:11.912 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:35.729 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:11.969 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:40.850 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:17.103 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:41.081 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:17.337 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:41.084 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:17.314 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:41.201 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:17.437 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:41.323 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-04T17:44:41.438 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.436+0000 7fc981752640 10 monclient: get_monmap_and_config 2024-05-04T17:44:41.438 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.436+0000 7fc981752640 10 monclient: build_initial_monmap 2024-05-04T17:44:41.438 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.436+0000 7fc981752640 10 monclient: monmap: 2024-05-04T17:44:41.438 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:44:41.438 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:41.438 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:44:41.438049+0000 2024-05-04T17:44:41.438 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:44:41.438049+0000 2024-05-04T17:44:41.438 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:41.439 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:41.439 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:41.439 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:41.439 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:41.439 INFO:teuthology.misc.health.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:41.439 INFO:teuthology.misc.health.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:41.439 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:41.440 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.436+0000 7fc981752640 1 Processor -- start 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.436+0000 7fc981752640 1 -- start start 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.436+0000 7fc981752640 10 monclient: init 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.437+0000 7fc981752640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.437+0000 7fc981752640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.437+0000 7fc981752640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc97c0a96e0 0x7fc97c1430d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.437+0000 7fc981752640 10 monclient(hunting): picked mon.noname-b con 0x7fc97c0a96e0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.437+0000 7fc981752640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc97c0a9ac0 0x7fc97c143610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.437+0000 7fc981752640 10 monclient(hunting): picked mon.noname-c con 0x7fc97c0a9ac0 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.438+0000 7fc981752640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc97c146130 addr v1:172.21.15.186:6789/0 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.438+0000 7fc981752640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc97c0ab910 con 0x7fc97c146130 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.438+0000 7fc981752640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.438+0000 7fc981752640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fc97c143b50 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.438+0000 7fc981752640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:41.441 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.438+0000 7fc981752640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc97c0abbf0 con 0x7fc97c0a96e0 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.438+0000 7fc981752640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.438+0000 7fc981752640 10 monclient(hunting): authenticate will time out at 1303.552690s 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.438+0000 7fc97affd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc97c0a96e0 0x7fc97c1430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.438+0000 7fc97a7fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc97c0a9ac0 0x7fc97c143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.439+0000 7fc97affd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc97c0a96e0 0x7fc97c1430d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:34422/0 (socket says 172.21.15.19:34422) 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.439+0000 7fc97affd640 1 -- 172.21.15.19:0/484912895 learned_addr learned my addr 172.21.15.19:0/484912895 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.439+0000 7fc97a7fc640 10 monclient(hunting): get_auth_request con 0x7fc97c0a9ac0 auth_method 0 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.439+0000 7fc97a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.439+0000 7fc97a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.439+0000 7fc97a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.439+0000 7fc97affd640 10 monclient(hunting): get_auth_request con 0x7fc97c0a96e0 auth_method 0 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.439+0000 7fc97affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:41.442 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.439+0000 7fc97affd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:41.443 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.439+0000 7fc97affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:41.443 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.440+0000 7fc97affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:41.443 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.440+0000 7fc97affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:41.443 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.440+0000 7fc97affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:41.443 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.440+0000 7fc97a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:41.443 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.440+0000 7fc97a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:41.444 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.440+0000 7fc97a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:41.444 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.442+0000 7fc97a7fc640 10 monclient(hunting): handle_auth_done global_id 4225 payload 274 2024-05-04T17:44:41.444 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.442+0000 7fc97a7fc640 1 -- 172.21.15.19:0/484912895 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc97c0a96e0 msgr2=0x7fc97c1430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:41.444 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.442+0000 7fc97a7fc640 1 --2- 172.21.15.19:0/484912895 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc97c0a96e0 0x7fc97c1430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:41.444 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.442+0000 7fc97a7fc640 1 -- 172.21.15.19:0/484912895 >> v1:172.21.15.186:6789/0 conn(0x7fc97c146130 legacy=0x7fc97c1465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:41.444 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.442+0000 7fc97a7fc640 10 monclient: _finish_hunting 0 2024-05-04T17:44:41.445 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.442+0000 7fc97a7fc640 1 monclient: found mon.noname-c 2024-05-04T17:44:41.445 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.442+0000 7fc97a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:41.445 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.442+0000 7fc97a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:44:41.445 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.442+0000 7fc97a7fc640 1 -- 172.21.15.19:0/484912895 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97c142bb0 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.448 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.447+0000 7fc97a7fc640 1 --2- 172.21.15.19:0/484912895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc97c0a9ac0 0x7fc97c143610 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9700097a0 tx=0x7fc97002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=470ef3d3b119051f server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:41.450 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:17.691 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:41.453 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.450+0000 7fc979ffb640 1 -- 172.21.15.19:0/484912895 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970035020 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.453 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.450+0000 7fc979ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:41.453 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.450+0000 7fc979ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.450+0000 7fc979ffb640 10 monclient: dump: 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.450+0000 7fc979ffb640 10 monclient: _finish_auth 0 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.450+0000 7fc979ffb640 10 monclient: _check_auth_tickets 2024-05-04T17:44:41.454 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.450+0000 7fc979ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc981752640 5 monclient: authenticate success, global_id 4225 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc979ffb640 1 -- 172.21.15.19:0/484912895 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc970031830 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc979ffb640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc981752640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc981752640 4 set_mon_vals no callback set 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc979ffb640 1 -- 172.21.15.19:0/484912895 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970031c00 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc979ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc979ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc979ffb640 10 monclient: dump: 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:41.455 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc981752640 10 monclient: shutdown 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc981752640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc981752640 1 -- 172.21.15.19:0/484912895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc97c0a9ac0 msgr2=0x7fc97c143610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.451+0000 7fc981752640 1 --2- 172.21.15.19:0/484912895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc97c0a9ac0 0x7fc97c143610 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9700097a0 tx=0x7fc97002b8d0 comp rx=0 tx=0).stop 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.452+0000 7fc981752640 1 -- 172.21.15.19:0/484912895 shutdown_connections 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.452+0000 7fc981752640 1 --2- 172.21.15.19:0/484912895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc97c0a9ac0 0x7fc97c143610 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.452+0000 7fc981752640 1 --2- 172.21.15.19:0/484912895 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc97c0a96e0 0x7fc97c1430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.452+0000 7fc981752640 1 -- 172.21.15.19:0/484912895 >> 172.21.15.19:0/484912895 conn(0x7fc97c0678d0 msgr2=0x7fc97c0b4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.452+0000 7fc981752640 1 -- 172.21.15.19:0/484912895 shutdown_connections 2024-05-04T17:44:41.456 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.452+0000 7fc981752640 1 -- 172.21.15.19:0/484912895 wait complete. 2024-05-04T17:44:41.460 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.453+0000 7fc981752640 10 monclient: build_initial_monmap 2024-05-04T17:44:41.460 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.453+0000 7fc981752640 10 monclient: monmap: 2024-05-04T17:44:41.460 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:44:41.460 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:41.460 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-05-04T17:44:41.460 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.453+0000 7fc981752640 1 librados: starting msgr at 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.453+0000 7fc981752640 1 librados: starting objecter 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.453+0000 7fc981752640 1 Processor -- start 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.453+0000 7fc981752640 1 -- start start 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.453+0000 7fc981752640 1 librados: setting wanted keys 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.453+0000 7fc981752640 1 librados: calling monclient init 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.453+0000 7fc981752640 10 monclient: init 2024-05-04T17:44:41.461 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.454+0000 7fc981752640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.454+0000 7fc981752640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.454+0000 7fc981752640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc97c0a96e0 0x7fc97c0b6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.454+0000 7fc981752640 10 monclient(hunting): picked mon.noname-a con 0x7fc97c0a96e0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.454+0000 7fc981752640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc97c0a9ac0 0x7fc97c13acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.454+0000 7fc981752640 10 monclient(hunting): picked mon.noname-b con 0x7fc97c0a9ac0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.454+0000 7fc981752640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc97c146130 0x7fc97c13b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.455+0000 7fc97a7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc97c0a9ac0 0x7fc97c13acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.455+0000 7fc981752640 10 monclient(hunting): picked mon.noname-c con 0x7fc97c146130 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.455+0000 7fc981752640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.455+0000 7fc981752640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c146cd0 con 0x7fc97c0a96e0 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.455+0000 7fc981752640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.455+0000 7fc981752640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c146e50 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.462 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.455+0000 7fc981752640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.455+0000 7fc981752640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc97c14d740 con 0x7fc97c146130 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.455+0000 7fc981752640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.455+0000 7fc981752640 10 monclient(hunting): authenticate will time out at 1303.570120s 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.455+0000 7fc97b7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc97c146130 0x7fc97c13b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.455+0000 7fc97affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc97c0a96e0 0x7fc97c0b6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97a7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc97c0a9ac0 0x7fc97c13acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:45006/0 (socket says 172.21.15.19:45006) 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97a7fc640 1 -- 172.21.15.19:0/945176551 learned_addr learned my addr 172.21.15.19:0/945176551 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97a7fc640 10 monclient(hunting): get_auth_request con 0x7fc97c0a9ac0 auth_method 0 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97b7fe640 10 monclient(hunting): get_auth_request con 0x7fc97c146130 auth_method 0 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:41.463 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97affd640 10 monclient(hunting): get_auth_request con 0x7fc97c0a96e0 auth_method 0 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97affd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.456+0000 7fc97affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.457+0000 7fc97affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.457+0000 7fc97b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.457+0000 7fc97b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:41.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.457+0000 7fc97b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:41.465 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.461+0000 7fc97a7fc640 10 monclient(hunting): handle_auth_done global_id 4228 payload 824 2024-05-04T17:44:41.465 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.461+0000 7fc97a7fc640 1 -- 172.21.15.19:0/945176551 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc97c146130 msgr2=0x7fc97c13b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:41.465 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.461+0000 7fc97a7fc640 1 --2- 172.21.15.19:0/945176551 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc97c146130 0x7fc97c13b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:41.465 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.461+0000 7fc97a7fc640 1 -- 172.21.15.19:0/945176551 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc97c0a96e0 msgr2=0x7fc97c0b6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:41.465 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.461+0000 7fc97a7fc640 1 --2- 172.21.15.19:0/945176551 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc97c0a96e0 0x7fc97c0b6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:41.465 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.461+0000 7fc97a7fc640 10 monclient: _finish_hunting 0 2024-05-04T17:44:41.465 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.461+0000 7fc97a7fc640 1 monclient: found mon.noname-b 2024-05-04T17:44:41.465 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.461+0000 7fc97a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:41.465 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.461+0000 7fc97a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:41.465 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.461+0000 7fc97a7fc640 1 -- 172.21.15.19:0/945176551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97c13a8a0 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.466 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.464+0000 7fc9637fe640 10 client.?.objecter ms_handle_connect 0x7fc97c0a9ac0 2024-05-04T17:44:41.466 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.464+0000 7fc9637fe640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:41.466 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.464+0000 7fc97a7fc640 1 --2- 172.21.15.19:0/945176551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc97c0a9ac0 0x7fc97c13acc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc97002bee0 tx=0x7fc9700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:41.471 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 1 -- 172.21.15.19:0/945176551 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970035040 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.471 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:41.471 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:41.471 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 10 monclient: dump: 2024-05-04T17:44:41.471 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:41.471 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:41.471 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:41.471 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:41.471 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 10 monclient: _finish_auth 0 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 10 monclient: _check_auth_tickets 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc981752640 5 monclient: authenticate success, global_id 4228 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc981752640 10 monclient: _renew_subs 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc981752640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc981752640 1 -- 172.21.15.19:0/945176551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc97c0b7bd0 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 1 -- 172.21.15.19:0/945176551 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc97003b900 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:41.472 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 1 -- 172.21.15.19:0/945176551 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970035040 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9637fe640 10 monclient: dump: 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:41.473 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc981752640 10 client.4228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:41.474 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc981752640 10 monclient: _renew_subs 2024-05-04T17:44:41.474 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc981752640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:41.474 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc981752640 1 -- 172.21.15.19:0/945176551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc97c13b740 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.474 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc981752640 1 librados: init done 2024-05-04T17:44:41.474 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.469+0000 7fc9797fa640 4 set_mon_vals no callback set 2024-05-04T17:44:41.474 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.470+0000 7fc981752640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:41.475 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.470+0000 7fc981752640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:41.475 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.470+0000 7fc981752640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:41.475 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.470+0000 7fc981752640 1 -- 172.21.15.19:0/945176551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc97c04f5f0 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.490 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:17.748 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:41.505 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.503+0000 7fc9637fe640 1 -- 172.21.15.19:0/945176551 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7fc970038020 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.506 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.504+0000 7fc9637fe640 1 --2- 172.21.15.19:0/945176551 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc96806d930 0x7fc96806fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:41.506 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.504+0000 7fc97affd640 1 --2- 172.21.15.19:0/945176551 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc96806d930 0x7fc96806fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:41.506 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.504+0000 7fc97affd640 10 monclient: get_auth_request con 0x7fc96806d930 auth_method 0 2024-05-04T17:44:41.507 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.505+0000 7fc9637fe640 10 client.4228.objecter ms_handle_connect 0x7fc96806d930 2024-05-04T17:44:41.507 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.505+0000 7fc97affd640 1 --2- 172.21.15.19:0/945176551 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc96806d930 0x7fc96806fdf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009bb0 tx=0x7fc96c009240 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:41.510 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.508+0000 7fc9637fe640 1 -- 172.21.15.19:0/945176551 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fc9700ab570 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.510 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.508+0000 7fc9637fe640 10 client.4228.objecter ms_dispatch 0x7fc97c000b90 osd_map(10..10 src has 1..10) 2024-05-04T17:44:41.510 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.508+0000 7fc9637fe640 3 client.4228.objecter handle_osd_map got epochs [10,10] > 0 2024-05-04T17:44:41.510 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.508+0000 7fc9637fe640 3 client.4228.objecter handle_osd_map decoding full epoch 10 2024-05-04T17:44:41.510 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.508+0000 7fc9637fe640 20 client.4228.objecter dump_active .. 0 homeless 2024-05-04T17:44:41.584 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:17.839 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:41.769 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:18.009 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:41.838 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.836+0000 7fc9637fe640 1 -- 172.21.15.19:0/945176551 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc970055080 con 0x7fc97c0a9ac0 2024-05-04T17:44:41.839 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.836+0000 7fc9637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:41.839 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:41.836+0000 7fc9637fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:42.169 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.167+0000 7fc981752640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:42.169 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.167+0000 7fc981752640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:42.169 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.167+0000 7fc981752640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:42.169 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.167+0000 7fc981752640 1 -- 172.21.15.19:0/945176551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc97c13d600 con 0x7fc97c0a9ac0 2024-05-04T17:44:42.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.186+0000 7fc9637fe640 1 -- 172.21.15.19:0/945176551 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fc97c0b7bd0 con 0x7fc97c0a9ac0 2024-05-04T17:44:42.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.186+0000 7fc9637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:42.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.186+0000 7fc9637fe640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:42.188 INFO:teuthology.misc.health.smithi019.stdout: 2024-05-04T17:44:42.188 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"18ced2df-a29f-46cc-b71a-1855bfb60c47","created":"2024-05-04T17:44:16.995866+0000","modified":"2024-05-04T17:44:23.464900+0000","last_up_change":"0.000000","last_in_change":"2024-05-04T17:44:23.464900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4b749ec0-22b6-458c-a7af-2ccace806ab7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3aa520fe-39c4-4fa2-860b-f4d0cc04ca97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16ed21c9-e5c3-4d65-b4d1-e4927a0fe919","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45aa2c0f-cfae-478c-a60c-4c1491e45525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb00f84f-d78c-461c-9843-73a9da4b80a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d07ad403-291f-4bde-a64a-5181fb79e544","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5110d1ad-5634-4c8e-9338-d6c4229b1d63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b307b905-5cc8-447b-8765-d888ed6f655b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:44:42.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.188+0000 7fc981752640 10 librados: watch_flush enter 2024-05-04T17:44:42.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.188+0000 7fc981752640 10 librados: watch_flush exit 2024-05-04T17:44:42.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.188+0000 7fc981752640 20 client.4228.objecter shutdown clearing up homeless session... 2024-05-04T17:44:42.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.188+0000 7fc981752640 10 client.4228.objecter successfully canceled tick 2024-05-04T17:44:42.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.188+0000 7fc981752640 1 -- 172.21.15.19:0/945176551 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc96806d930 msgr2=0x7fc96806fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:42.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.188+0000 7fc981752640 1 --2- 172.21.15.19:0/945176551 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc96806d930 0x7fc96806fdf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009bb0 tx=0x7fc96c009240 comp rx=0 tx=0).stop 2024-05-04T17:44:42.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 10 monclient: shutdown 2024-05-04T17:44:42.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:42.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 1 -- 172.21.15.19:0/945176551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc97c0a9ac0 msgr2=0x7fc97c13acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:42.192 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 1 --2- 172.21.15.19:0/945176551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc97c0a9ac0 0x7fc97c13acc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc97002bee0 tx=0x7fc9700046c0 comp rx=0 tx=0).stop 2024-05-04T17:44:42.192 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 1 -- 172.21.15.19:0/945176551 shutdown_connections 2024-05-04T17:44:42.192 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 1 --2- 172.21.15.19:0/945176551 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc96806d930 0x7fc96806fdf0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:42.192 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 1 --2- 172.21.15.19:0/945176551 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc97c146130 0x7fc97c13b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:42.192 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 1 --2- 172.21.15.19:0/945176551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc97c0a9ac0 0x7fc97c13acc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:42.192 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 1 --2- 172.21.15.19:0/945176551 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc97c0a96e0 0x7fc97c0b6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:42.192 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 1 -- 172.21.15.19:0/945176551 >> 172.21.15.19:0/945176551 conn(0x7fc97c0678d0 msgr2=0x7fc97c142fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:42.192 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 1 -- 172.21.15.19:0/945176551 shutdown_connections 2024-05-04T17:44:42.192 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 1 -- 172.21.15.19:0/945176551 wait complete. 2024-05-04T17:44:42.192 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:42.189+0000 7fc981752640 1 librados: shutdown 2024-05-04T17:44:42.201 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-04T17:44:45.986 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:22.242 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:45.990 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:22.247 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:46.140 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:22.393 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:46.144 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:22.397 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:46.149 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:22.379 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:46.155 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:22.385 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:46.198 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:22.457 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:46.205 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:22.463 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:46.454 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:22.690 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:46.461 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:22.697 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:46.921 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:23.162 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:46.925 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:23.166 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:47.157 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:23.412 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:47.161 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:23.416 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:47.337 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:23.577 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:47.343 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:23.583 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:49.008 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-04T17:44:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.141+0000 7f2bcc85f640 10 monclient: get_monmap_and_config 2024-05-04T17:44:49.143 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.141+0000 7f2bcc85f640 10 monclient: build_initial_monmap 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.141+0000 7f2bcc85f640 10 monclient: monmap: 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:44:49.143089+0000 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:44:49.143089+0000 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.141+0000 7f2bcc85f640 1 Processor -- start 2024-05-04T17:44:49.144 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.141+0000 7f2bcc85f640 1 -- start start 2024-05-04T17:44:49.145 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.141+0000 7f2bcc85f640 10 monclient: init 2024-05-04T17:44:49.147 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:49.147 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-05-04T17:44:49.147 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2bc80a96e0 0x7f2bc81430d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:49.147 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 10 monclient(hunting): picked mon.noname-b con 0x7f2bc80a96e0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:49.148 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2bc80a9ac0 0x7f2bc8143610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:49.148 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 10 monclient(hunting): picked mon.noname-c con 0x7f2bc80a9ac0 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:49.148 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2bc8146130 addr v1:172.21.15.19:6789/0 2024-05-04T17:44:49.148 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bc80ab910 con 0x7f2bc8146130 2024-05-04T17:44:49.148 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:49.148 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f2bc8143b50 con 0x7f2bc80a9ac0 2024-05-04T17:44:49.148 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:49.148 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f2bc80abbf0 con 0x7f2bc80a96e0 2024-05-04T17:44:49.148 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:49.148 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.142+0000 7f2bcc85f640 10 monclient(hunting): authenticate will time out at 1311.257349s 2024-05-04T17:44:49.148 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.143+0000 7f2bc659c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2bc80a96e0 0x7f2bc81430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:49.149 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.143+0000 7f2bc6d9d640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f2bc8146130 0x7f2bc81465c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:38032/0 (socket says 172.21.15.19:38032) 2024-05-04T17:44:49.149 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.143+0000 7f2bc6d9d640 1 -- 172.21.15.19:0/851559368 learned_addr learned my addr 172.21.15.19:0/851559368 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:44:49.149 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.143+0000 7f2bc5d9b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2bc80a9ac0 0x7f2bc8143610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:49.149 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.143+0000 7f2bc659c640 10 monclient(hunting): get_auth_request con 0x7f2bc80a96e0 auth_method 0 2024-05-04T17:44:49.149 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.143+0000 7f2bc659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:49.149 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.143+0000 7f2bc659c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:49.149 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.143+0000 7f2bc659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:49.149 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.144+0000 7f2bc5d9b640 10 monclient(hunting): get_auth_request con 0x7f2bc80a9ac0 auth_method 0 2024-05-04T17:44:49.149 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.144+0000 7f2bc5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:49.150 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.144+0000 7f2bc5d9b640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:49.150 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.144+0000 7f2bc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:49.150 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.144+0000 7f2bc5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:49.150 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.144+0000 7f2bc5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:49.150 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.144+0000 7f2bc5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:49.150 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.145+0000 7f2bc659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:49.150 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.145+0000 7f2bc659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:49.150 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.145+0000 7f2bc659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.147+0000 7f2bc5d9b640 10 monclient(hunting): handle_auth_done global_id 4231 payload 274 2024-05-04T17:44:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.147+0000 7f2bc5d9b640 1 -- 172.21.15.19:0/851559368 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2bc80a96e0 msgr2=0x7f2bc81430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.147+0000 7f2bc5d9b640 1 --2- 172.21.15.19:0/851559368 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2bc80a96e0 0x7f2bc81430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.147+0000 7f2bc5d9b640 1 -- 172.21.15.19:0/851559368 >> v1:172.21.15.19:6789/0 conn(0x7f2bc8146130 legacy=0x7f2bc81465c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.147+0000 7f2bc5d9b640 10 monclient: _finish_hunting 0 2024-05-04T17:44:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.147+0000 7f2bc5d9b640 1 monclient: found mon.noname-c 2024-05-04T17:44:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.147+0000 7f2bc5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.147+0000 7f2bc5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:44:49.151 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.147+0000 7f2bc5d9b640 1 -- 172.21.15.19:0/851559368 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc8142bb0 con 0x7f2bc80a9ac0 2024-05-04T17:44:49.152 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.149+0000 7f2bc5d9b640 1 --2- 172.21.15.19:0/851559368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bc80a9ac0 0x7f2bc8143610 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0097a0 tx=0x7f2bbc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e033eea0e222d401 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:49.155 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 1 -- 172.21.15.19:0/851559368 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc035020 con 0x7f2bc80a9ac0 2024-05-04T17:44:49.155 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:49.155 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:44:49.155 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 10 monclient: dump: 2024-05-04T17:44:49.156 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:49.156 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:49.156 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:49.156 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:49.156 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:49.156 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:49.156 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:49.156 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:49.156 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:49.157 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:49.157 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 10 monclient: _finish_auth 0 2024-05-04T17:44:49.157 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 10 monclient: _check_auth_tickets 2024-05-04T17:44:49.157 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:49.157 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 1 -- 172.21.15.19:0/851559368 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bbc031810 con 0x7f2bc80a9ac0 2024-05-04T17:44:49.157 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:49.157 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 1 -- 172.21.15.19:0/851559368 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc031be0 con 0x7f2bc80a9ac0 2024-05-04T17:44:49.157 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:49.158 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:49.158 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bc559a640 10 monclient: dump: 2024-05-04T17:44:49.158 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:49.158 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:49.158 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:49.158 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:49.158 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:49.158 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:49.159 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:49.159 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:49.159 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:49.159 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:49.159 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bcc85f640 5 monclient: authenticate success, global_id 4231 2024-05-04T17:44:49.159 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bcc85f640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:49.159 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.153+0000 7f2bcc85f640 4 set_mon_vals no callback set 2024-05-04T17:44:49.175 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.154+0000 7f2bcc85f640 10 monclient: shutdown 2024-05-04T17:44:49.175 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.154+0000 7f2bcc85f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:49.176 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.154+0000 7f2bcc85f640 1 -- 172.21.15.19:0/851559368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bc80a9ac0 msgr2=0x7f2bc8143610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:49.176 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.154+0000 7f2bcc85f640 1 --2- 172.21.15.19:0/851559368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bc80a9ac0 0x7f2bc8143610 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0097a0 tx=0x7f2bbc02b8b0 comp rx=0 tx=0).stop 2024-05-04T17:44:49.176 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.154+0000 7f2bcc85f640 1 -- 172.21.15.19:0/851559368 shutdown_connections 2024-05-04T17:44:49.176 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.154+0000 7f2bcc85f640 1 --2- 172.21.15.19:0/851559368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bc80a9ac0 0x7f2bc8143610 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:49.176 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.154+0000 7f2bcc85f640 1 --2- 172.21.15.19:0/851559368 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2bc80a96e0 0x7f2bc81430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:49.177 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.154+0000 7f2bcc85f640 1 -- 172.21.15.19:0/851559368 >> 172.21.15.19:0/851559368 conn(0x7f2bc80678d0 msgr2=0x7f2bc80b4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:49.177 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.154+0000 7f2bcc85f640 1 -- 172.21.15.19:0/851559368 shutdown_connections 2024-05-04T17:44:49.177 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.154+0000 7f2bcc85f640 1 -- 172.21.15.19:0/851559368 wait complete. 2024-05-04T17:44:49.177 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.155+0000 7f2bcc85f640 10 monclient: build_initial_monmap 2024-05-04T17:44:49.177 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.155+0000 7f2bcc85f640 10 monclient: monmap: 2024-05-04T17:44:49.177 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:44:49.178 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:49.178 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-05-04T17:44:49.178 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-05-04T17:44:49.178 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:49.178 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:49.178 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:49.179 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:49.179 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:49.179 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:49.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.156+0000 7f2bcc85f640 1 librados: starting msgr at 2024-05-04T17:44:49.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.156+0000 7f2bcc85f640 1 librados: starting objecter 2024-05-04T17:44:49.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.156+0000 7f2bcc85f640 1 Processor -- start 2024-05-04T17:44:49.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.156+0000 7f2bcc85f640 1 -- start start 2024-05-04T17:44:49.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.156+0000 7f2bcc85f640 1 librados: setting wanted keys 2024-05-04T17:44:49.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.156+0000 7f2bcc85f640 1 librados: calling monclient init 2024-05-04T17:44:49.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.156+0000 7f2bcc85f640 10 monclient: init 2024-05-04T17:44:49.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.157+0000 7f2bcc85f640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:49.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.157+0000 7f2bcc85f640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T17:44:49.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.157+0000 7f2bcc85f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bc80a96e0 0x7f2bc813ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:49.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.157+0000 7f2bcc85f640 10 monclient(hunting): picked mon.noname-b con 0x7f2bc80a96e0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:49.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.157+0000 7f2bcc85f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bc80a9ac0 0x7f2bc813f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:49.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.158+0000 7f2bcc85f640 10 monclient(hunting): picked mon.noname-a con 0x7f2bc80a9ac0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.158+0000 7f2bcc85f640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2bc8146130 0x7f2bc80b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.158+0000 7f2bcc85f640 10 monclient(hunting): picked mon.noname-c con 0x7f2bc8146130 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.158+0000 7f2bc5d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bc80a9ac0 0x7f2bc813f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.158+0000 7f2bcc85f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.158+0000 7f2bc659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bc80a96e0 0x7f2bc813ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc6d9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2bc8146130 0x7f2bc80b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc5d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bc80a9ac0 0x7f2bc813f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:42896/0 (socket says 172.21.15.19:42896) 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc5d9b640 1 -- 172.21.15.19:0/4274145768 learned_addr learned my addr 172.21.15.19:0/4274145768 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.158+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc8146cd0 con 0x7f2bc80a9ac0 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bcc85f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc8146e50 con 0x7f2bc80a96e0 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bcc85f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f2bc814d740 con 0x7f2bc8146130 2024-05-04T17:44:49.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bcc85f640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bcc85f640 10 monclient(hunting): authenticate will time out at 1311.274209s 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc5d9b640 10 monclient(hunting): get_auth_request con 0x7f2bc80a9ac0 auth_method 0 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc5d9b640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc659c640 10 monclient(hunting): get_auth_request con 0x7f2bc80a96e0 auth_method 0 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc659c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc659c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc6d9d640 10 monclient(hunting): get_auth_request con 0x7f2bc8146130 auth_method 0 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc6d9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.159+0000 7f2bc6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.160+0000 7f2bc659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:49.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.160+0000 7f2bc659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.160+0000 7f2bc659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.160+0000 7f2bc5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.160+0000 7f2bc5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.160+0000 7f2bc5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.160+0000 7f2bc6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.160+0000 7f2bc6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.160+0000 7f2bc6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.162+0000 7f2bc659c640 10 monclient(hunting): handle_auth_done global_id 4234 payload 824 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.162+0000 7f2bc659c640 1 -- 172.21.15.19:0/4274145768 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2bc8146130 msgr2=0x7f2bc80b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.162+0000 7f2bc659c640 1 --2- 172.21.15.19:0/4274145768 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2bc8146130 0x7f2bc80b64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.162+0000 7f2bc659c640 1 -- 172.21.15.19:0/4274145768 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bc80a9ac0 msgr2=0x7f2bc813f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.162+0000 7f2bc659c640 1 --2- 172.21.15.19:0/4274145768 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bc80a9ac0 0x7f2bc813f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.162+0000 7f2bc659c640 10 monclient: _finish_hunting 0 2024-05-04T17:44:49.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.162+0000 7f2bc659c640 1 monclient: found mon.noname-b 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.162+0000 7f2bc659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.162+0000 7f2bc659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.162+0000 7f2bc659c640 1 -- 172.21.15.19:0/4274145768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc813a550 con 0x7f2bc80a96e0 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.165+0000 7f2bc659c640 1 --2- 172.21.15.19:0/4274145768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bc80a96e0 0x7f2bc813ef10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800e850 tx=0x7f2bb800ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.165+0000 7f2bb6ffd640 10 client.?.objecter ms_handle_connect 0x7f2bc80a96e0 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.165+0000 7f2bb6ffd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 1 -- 172.21.15.19:0/4274145768 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb8017070 con 0x7f2bc80a96e0 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 10 monclient: dump: 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:49.186 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 10 monclient: _finish_auth 0 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 10 monclient: _check_auth_tickets 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 1 -- 172.21.15.19:0/4274145768 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bb800c940 con 0x7f2bc80a96e0 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bcc85f640 5 monclient: authenticate success, global_id 4234 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bcc85f640 10 monclient: _renew_subs 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bcc85f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc813a830 con 0x7f2bc80a96e0 2024-05-04T17:44:49.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:49.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 1 -- 172.21.15.19:0/4274145768 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb800cd10 con 0x7f2bc80a96e0 2024-05-04T17:44:49.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:49.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bcc85f640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:49.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:49.189 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bb6ffd640 10 monclient: dump: 2024-05-04T17:44:49.189 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:49.189 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:49.190 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:49.190 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:49.190 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:49.190 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:49.190 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:49.191 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:49.191 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:49.191 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bcc85f640 10 monclient: _renew_subs 2024-05-04T17:44:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bcc85f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc8140050 con 0x7f2bc80a96e0 2024-05-04T17:44:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bcc85f640 1 librados: init done 2024-05-04T17:44:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.169+0000 7f2bc4d99640 4 set_mon_vals no callback set 2024-05-04T17:44:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.170+0000 7f2bcc85f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.170+0000 7f2bcc85f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.170+0000 7f2bcc85f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:49.191 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.170+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bc804f5f0 con 0x7f2bc80a96e0 2024-05-04T17:44:49.313 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.311+0000 7f2bb6ffd640 1 -- 172.21.15.19:0/4274145768 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f2bb801a070 con 0x7f2bc80a96e0 2024-05-04T17:44:49.314 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.312+0000 7f2bb6ffd640 1 --2- 172.21.15.19:0/4274145768 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f2ba806d780 0x7f2ba806fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:49.314 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.312+0000 7f2bc5d9b640 1 --2- 172.21.15.19:0/4274145768 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f2ba806d780 0x7f2ba806fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:49.314 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.312+0000 7f2bc5d9b640 10 monclient: get_auth_request con 0x7f2ba806d780 auth_method 0 2024-05-04T17:44:49.314 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.313+0000 7f2bb6ffd640 10 client.4234.objecter ms_handle_connect 0x7f2ba806d780 2024-05-04T17:44:49.314 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.313+0000 7f2bc5d9b640 1 --2- 172.21.15.19:0/4274145768 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f2ba806d780 0x7f2ba806fc40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8134c20 tx=0x7f2bbc008030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:49.500 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.498+0000 7f2bb6ffd640 1 -- 172.21.15.19:0/4274145768 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f2bb80584d0 con 0x7f2bc80a96e0 2024-05-04T17:44:49.500 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.498+0000 7f2bb6ffd640 10 client.4234.objecter ms_dispatch 0x7f2bc8000b90 osd_map(10..10 src has 1..10) 2024-05-04T17:44:49.501 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.498+0000 7f2bb6ffd640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-05-04T17:44:49.501 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.498+0000 7f2bb6ffd640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-05-04T17:44:49.501 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.498+0000 7f2bb6ffd640 20 client.4234.objecter dump_active .. 0 homeless 2024-05-04T17:44:49.501 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.498+0000 7f2bb6ffd640 1 -- 172.21.15.19:0/4274145768 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2bb8059840 con 0x7f2bc80a96e0 2024-05-04T17:44:49.501 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.498+0000 7f2bb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:49.501 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.498+0000 7f2bb6ffd640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:49.770 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.768+0000 7f2bcc85f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:49.770 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.768+0000 7f2bcc85f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:49.770 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.768+0000 7f2bcc85f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:49.770 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.768+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bc8073100 con 0x7f2bc80a96e0 2024-05-04T17:44:49.797 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.795+0000 7f2bb6ffd640 1 -- 172.21.15.19:0/4274145768 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f2bb805db40 con 0x7f2bc80a96e0 2024-05-04T17:44:49.797 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.795+0000 7f2bb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:49.797 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.795+0000 7f2bb6ffd640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:49.798 INFO:teuthology.misc.health.smithi019.stdout: 2024-05-04T17:44:49.798 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"18ced2df-a29f-46cc-b71a-1855bfb60c47","created":"2024-05-04T17:44:16.995866+0000","modified":"2024-05-04T17:44:23.464900+0000","last_up_change":"0.000000","last_in_change":"2024-05-04T17:44:23.464900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4b749ec0-22b6-458c-a7af-2ccace806ab7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3aa520fe-39c4-4fa2-860b-f4d0cc04ca97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16ed21c9-e5c3-4d65-b4d1-e4927a0fe919","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45aa2c0f-cfae-478c-a60c-4c1491e45525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb00f84f-d78c-461c-9843-73a9da4b80a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d07ad403-291f-4bde-a64a-5181fb79e544","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5110d1ad-5634-4c8e-9338-d6c4229b1d63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b307b905-5cc8-447b-8765-d888ed6f655b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:44:49.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 10 librados: watch_flush enter 2024-05-04T17:44:49.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 10 librados: watch_flush exit 2024-05-04T17:44:49.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 20 client.4234.objecter shutdown clearing up homeless session... 2024-05-04T17:44:49.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 10 client.4234.objecter successfully canceled tick 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f2ba806d780 msgr2=0x7f2ba806fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 --2- 172.21.15.19:0/4274145768 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f2ba806d780 0x7f2ba806fc40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8134c20 tx=0x7f2bbc008030 comp rx=0 tx=0).stop 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 10 monclient: shutdown 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bc80a96e0 msgr2=0x7f2bc813ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 --2- 172.21.15.19:0/4274145768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bc80a96e0 0x7f2bc813ef10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800e850 tx=0x7f2bb800ed10 comp rx=0 tx=0).stop 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 shutdown_connections 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 --2- 172.21.15.19:0/4274145768 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f2ba806d780 0x7f2ba806fc40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 --2- 172.21.15.19:0/4274145768 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f2bc8146130 0x7f2bc80b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 --2- 172.21.15.19:0/4274145768 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2bc80a9ac0 0x7f2bc813f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 --2- 172.21.15.19:0/4274145768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bc80a96e0 0x7f2bc813ef10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 >> 172.21.15.19:0/4274145768 conn(0x7f2bc80678d0 msgr2=0x7f2bc8142fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 shutdown_connections 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 -- 172.21.15.19:0/4274145768 wait complete. 2024-05-04T17:44:49.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:49.798+0000 7f2bcc85f640 1 librados: shutdown 2024-05-04T17:44:49.813 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-04T17:44:50.146 INFO:tasks.ceph.osd.5.smithi186.stderr:==00:00:00:26.402 33086== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:50.221 INFO:tasks.ceph.osd.7.smithi186.stderr:==00:00:00:26.451 33088== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:50.420 INFO:tasks.ceph.osd.4.smithi186.stderr:==00:00:00:26.678 33085== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:50.448 INFO:tasks.ceph.osd.6.smithi186.stderr:==00:00:00:26.701 33087== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:50.671 INFO:tasks.ceph.osd.2.smithi019.stderr:==00:00:00:26.907 33122== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:51.451 INFO:tasks.ceph.osd.3.smithi019.stderr:==00:00:00:27.692 33120== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:51.472 INFO:tasks.ceph.osd.0.smithi019.stderr:==00:00:00:27.712 33121== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:51.505 INFO:tasks.ceph.osd.1.smithi019.stderr:==00:00:00:27.760 33119== Warning: unimplemented fcntl command: 1038 2024-05-04T17:44:56.620 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-04T17:44:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.770+0000 7f81a5997640 10 monclient: get_monmap_and_config 2024-05-04T17:44:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.770+0000 7f81a5997640 10 monclient: build_initial_monmap 2024-05-04T17:44:56.771 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.770+0000 7f81a5997640 10 monclient: monmap: 2024-05-04T17:44:56.772 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:44:56.772 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:56.772 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:44:56.771671+0000 2024-05-04T17:44:56.772 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:44:56.771671+0000 2024-05-04T17:44:56.772 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:56.772 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:56.772 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:44:56.772 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:44:56.772 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:44:56.772 INFO:teuthology.misc.health.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:44:56.772 INFO:teuthology.misc.health.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:44:56.772 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:56.775 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.771+0000 7f81a5997640 1 Processor -- start 2024-05-04T17:44:56.775 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.771+0000 7f81a5997640 1 -- start start 2024-05-04T17:44:56.775 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.771+0000 7f81a5997640 10 monclient: init 2024-05-04T17:44:56.775 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.772+0000 7f81a5997640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:56.775 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.772+0000 7f81a5997640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-05-04T17:44:56.775 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.772+0000 7f81a5997640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81a00a96e0 addr v1:172.21.15.186:6789/0 2024-05-04T17:44:56.775 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.772+0000 7f81a5997640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f81a00a9ac0 0x7f81a013f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:56.775 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.772+0000 7f81a5997640 10 monclient(hunting): picked mon.noname-a con 0x7f81a00a9ac0 addr v2:172.21.15.19:3300/0 2024-05-04T17:44:56.775 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.772+0000 7f81a5997640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f81a01437e0 0x7f81a013fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:56.775 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.773+0000 7f81a5997640 10 monclient(hunting): picked mon.noname-c con 0x7f81a01437e0 addr v2:172.21.15.186:3300/0 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.773+0000 7f81a5997640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81a00abbf0 con 0x7f81a00a96e0 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.773+0000 7f81a5997640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.773+0000 7f81a5997640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f81a00ab9d0 con 0x7f81a00a9ac0 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.773+0000 7f81a5997640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.773+0000 7f81a5997640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f81a0142bb0 con 0x7f81a01437e0 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.773+0000 7f81a5997640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.773+0000 7f81a5997640 10 monclient(hunting): authenticate will time out at 1318.887369s 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.773+0000 7f819effd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f81a00a96e0 0x7f81a01430d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:43514/0 (socket says 172.21.15.19:43514) 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.773+0000 7f819effd640 1 -- 172.21.15.19:0/4204059031 learned_addr learned my addr 172.21.15.19:0/4204059031 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.774+0000 7f819e7fc640 1 --2- 172.21.15.19:0/4204059031 >> v2:172.21.15.19:3300/0 conn(0x7f81a00a9ac0 0x7f81a013f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.774+0000 7f819e7fc640 10 monclient(hunting): get_auth_request con 0x7f81a00a9ac0 auth_method 0 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.774+0000 7f819e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.774+0000 7f819e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:56.776 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.774+0000 7f819e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:56.777 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.774+0000 7f819f7fe640 1 --2- 172.21.15.19:0/4204059031 >> v2:172.21.15.186:3300/0 conn(0x7f81a01437e0 0x7f81a013fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:56.779 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.775+0000 7f819e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:56.779 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.775+0000 7f819e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:56.779 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.775+0000 7f819e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:56.779 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.775+0000 7f819f7fe640 10 monclient(hunting): get_auth_request con 0x7f81a01437e0 auth_method 0 2024-05-04T17:44:56.779 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.775+0000 7f819f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:56.779 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.775+0000 7f819f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:56.779 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.775+0000 7f819f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:56.779 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.776+0000 7f819f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:56.779 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.776+0000 7f819f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:56.779 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.776+0000 7f819f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:56.779 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.777+0000 7f819e7fc640 10 monclient(hunting): handle_auth_done global_id 4203 payload 274 2024-05-04T17:44:56.779 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.777+0000 7f819e7fc640 1 -- 172.21.15.19:0/4204059031 >> v2:172.21.15.186:3300/0 conn(0x7f81a01437e0 msgr2=0x7f81a013fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:44:56.780 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.777+0000 7f819e7fc640 1 --2- 172.21.15.19:0/4204059031 >> v2:172.21.15.186:3300/0 conn(0x7f81a01437e0 0x7f81a013fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:56.780 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.777+0000 7f819e7fc640 1 -- 172.21.15.19:0/4204059031 >> v1:172.21.15.186:6789/0 conn(0x7f81a00a96e0 legacy=0x7f81a01430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:56.780 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.777+0000 7f819e7fc640 10 monclient: _finish_hunting 0 2024-05-04T17:44:56.780 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.777+0000 7f819e7fc640 1 monclient: found mon.noname-a 2024-05-04T17:44:56.780 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.777+0000 7f819e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:56.780 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.777+0000 7f819e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:44:56.780 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.777+0000 7f819e7fc640 1 -- 172.21.15.19:0/4204059031 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a00b2ec0 con 0x7f81a00a9ac0 2024-05-04T17:44:56.781 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.780+0000 7f819e7fc640 1 --2- 172.21.15.19:0/4204059031 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81a00a9ac0 0x7f81a013f720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f81940097a0 tx=0x7f819402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2967e6b223516e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:56.789 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 1 -- 172.21.15.19:0/4204059031 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8194035020 con 0x7f81a00a9ac0 2024-05-04T17:44:56.789 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 10 monclient: dump: 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:56.790 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 10 monclient: _finish_auth 0 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 10 monclient: _check_auth_tickets 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 1 -- 172.21.15.19:0/4204059031 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8194031830 con 0x7f81a00a9ac0 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f81a5997640 5 monclient: authenticate success, global_id 4203 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f81a5997640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f81a5997640 10 monclient: get_monmap_and_config success 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f81a5997640 4 set_mon_vals no callback set 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 1 -- 172.21.15.19:0/4204059031 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8194031c00 con 0x7f81a00a9ac0 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.786+0000 7f819dffb640 10 monclient: dump: 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:56.791 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.787+0000 7f81a5997640 10 monclient: shutdown 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.787+0000 7f81a5997640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.787+0000 7f81a5997640 1 -- 172.21.15.19:0/4204059031 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81a00a9ac0 msgr2=0x7f81a013f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.787+0000 7f81a5997640 1 --2- 172.21.15.19:0/4204059031 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81a00a9ac0 0x7f81a013f720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f81940097a0 tx=0x7f819402b8d0 comp rx=0 tx=0).stop 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.787+0000 7f81a5997640 1 -- 172.21.15.19:0/4204059031 shutdown_connections 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.787+0000 7f81a5997640 1 --2- 172.21.15.19:0/4204059031 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81a00a9ac0 0x7f81a013f720 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:56.792 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.787+0000 7f81a5997640 1 --2- 172.21.15.19:0/4204059031 >> v2:172.21.15.186:3300/0 conn(0x7f81a01437e0 0x7f81a013fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.787+0000 7f81a5997640 1 -- 172.21.15.19:0/4204059031 >> 172.21.15.19:0/4204059031 conn(0x7f81a00678d0 msgr2=0x7f81a00b4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.788+0000 7f81a5997640 1 -- 172.21.15.19:0/4204059031 shutdown_connections 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.788+0000 7f81a5997640 1 -- 172.21.15.19:0/4204059031 wait complete. 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.789+0000 7f81a5997640 10 monclient: build_initial_monmap 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.789+0000 7f81a5997640 10 monclient: monmap: 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:44:56.793 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:56.799 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.789+0000 7f81a5997640 1 librados: starting msgr at 2024-05-04T17:44:56.799 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.789+0000 7f81a5997640 1 librados: starting objecter 2024-05-04T17:44:56.799 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.789+0000 7f81a5997640 1 Processor -- start 2024-05-04T17:44:56.799 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.790+0000 7f81a5997640 1 -- start start 2024-05-04T17:44:56.799 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.790+0000 7f81a5997640 1 librados: setting wanted keys 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.790+0000 7f81a5997640 1 librados: calling monclient init 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.790+0000 7f81a5997640 10 monclient: init 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.790+0000 7f81a5997640 10 monclient: _reopen_session rank -1 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.790+0000 7f81a5997640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.790+0000 7f81a5997640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f81a00a96e0 0x7f81a0144a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 10 monclient(hunting): picked mon.noname-c con 0x7f81a00a96e0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a00a9ac0 0x7f81a01472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 10 monclient(hunting): picked mon.noname-b con 0x7f81a00a9ac0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81a01437e0 0x7f81a01457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 10 monclient(hunting): picked mon.noname-a con 0x7f81a01437e0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f81a0140320 con 0x7f81a01437e0 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:56.800 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f81a01401a0 con 0x7f81a00a9ac0 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 10 monclient(hunting): start opening mon connection 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f81a01404a0 con 0x7f81a00a96e0 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 10 monclient(hunting): _renew_subs 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f81a5997640 10 monclient(hunting): authenticate will time out at 1318.905209s 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f819e7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a00a9ac0 0x7f81a01472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f819effd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f81a00a96e0 0x7f81a0144a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.791+0000 7f819f7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81a01437e0 0x7f81a01457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819e7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a00a9ac0 0x7f81a01472e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:35292/0 (socket says 172.21.15.19:35292) 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819e7fc640 1 -- 172.21.15.19:0/1290319004 learned_addr learned my addr 172.21.15.19:0/1290319004 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819e7fc640 10 monclient(hunting): get_auth_request con 0x7f81a00a9ac0 auth_method 0 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819effd640 10 monclient(hunting): get_auth_request con 0x7f81a00a96e0 auth_method 0 2024-05-04T17:44:56.801 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819effd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819f7fe640 10 monclient(hunting): get_auth_request con 0x7f81a01437e0 auth_method 0 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.792+0000 7f819f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.793+0000 7f819e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.793+0000 7f819e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.793+0000 7f819e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.793+0000 7f819effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.793+0000 7f819effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.793+0000 7f819effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.793+0000 7f819f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:44:56.802 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.793+0000 7f819f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:44:56.803 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.793+0000 7f819f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:44:56.803 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.797+0000 7f819e7fc640 10 monclient(hunting): handle_auth_done global_id 4243 payload 824 2024-05-04T17:44:56.803 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.797+0000 7f819e7fc640 1 -- 172.21.15.19:0/1290319004 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f81a00a96e0 msgr2=0x7f81a0144a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:56.803 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.797+0000 7f819e7fc640 1 --2- 172.21.15.19:0/1290319004 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f81a00a96e0 0x7f81a0144a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:56.803 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.797+0000 7f819e7fc640 1 -- 172.21.15.19:0/1290319004 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81a01437e0 msgr2=0x7f81a01457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:56.803 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.797+0000 7f819e7fc640 1 --2- 172.21.15.19:0/1290319004 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81a01437e0 0x7f81a01457c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:56.803 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.797+0000 7f819e7fc640 10 monclient: _finish_hunting 0 2024-05-04T17:44:56.803 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.797+0000 7f819e7fc640 1 monclient: found mon.noname-b 2024-05-04T17:44:56.803 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.797+0000 7f819e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:44:56.803 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.797+0000 7f819e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:44:56.803 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.797+0000 7f819e7fc640 1 -- 172.21.15.19:0/1290319004 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a01452a0 con 0x7f81a00a9ac0 2024-05-04T17:44:56.804 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.800+0000 7f819e7fc640 1 --2- 172.21.15.19:0/1290319004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a00a9ac0 0x7f81a01472e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f819402bee0 tx=0x7f81940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:56.804 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.801+0000 7f8187fff640 10 client.?.objecter ms_handle_connect 0x7f81a00a9ac0 2024-05-04T17:44:56.804 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.801+0000 7f8187fff640 10 client.?.objecter resend_mon_ops 2024-05-04T17:44:56.809 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 1 -- 172.21.15.19:0/1290319004 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8194035040 con 0x7f81a00a9ac0 2024-05-04T17:44:56.809 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:56.809 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:44:56.809 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 10 monclient: dump: 2024-05-04T17:44:56.809 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:56.809 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:56.809 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:56.809 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:56.809 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:56.809 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 10 monclient: _finish_auth 0 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 10 monclient: _check_auth_tickets 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 1 -- 172.21.15.19:0/1290319004 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f819403b920 con 0x7f81a00a9ac0 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 10 monclient: handle_config config(0 keys) 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 1 -- 172.21.15.19:0/1290319004 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819403bd10 con 0x7f81a00a9ac0 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f8187fff640 10 monclient: dump: 2024-05-04T17:44:56.810 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f81a5997640 5 monclient: authenticate success, global_id 4243 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f81a5997640 10 monclient: _renew_subs 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f81a5997640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f81a5997640 1 -- 172.21.15.19:0/1290319004 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81a0144fa0 con 0x7f81a00a9ac0 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f819d7fa640 4 set_mon_vals no callback set 2024-05-04T17:44:56.811 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f81a5997640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:44:56.812 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f81a5997640 10 monclient: _renew_subs 2024-05-04T17:44:56.812 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f81a5997640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:56.812 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f81a5997640 1 -- 172.21.15.19:0/1290319004 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81a00b1c80 con 0x7f81a00a9ac0 2024-05-04T17:44:56.812 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.806+0000 7f81a5997640 1 librados: init done 2024-05-04T17:44:56.812 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.807+0000 7f81a5997640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:56.812 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.807+0000 7f81a5997640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:56.812 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.808+0000 7f81a5997640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:56.812 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:56.808+0000 7f81a5997640 1 -- 172.21.15.19:0/1290319004 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81a004f5f0 con 0x7f81a00a9ac0 2024-05-04T17:44:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.252+0000 7f8187fff640 1 -- 172.21.15.19:0/1290319004 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f8194038020 con 0x7f81a00a9ac0 2024-05-04T17:44:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.253+0000 7f8187fff640 1 --2- 172.21.15.19:0/1290319004 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f818c06d730 0x7f818c06fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:44:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.253+0000 7f819effd640 1 --2- 172.21.15.19:0/1290319004 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f818c06d730 0x7f818c06fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:44:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.253+0000 7f819effd640 10 monclient: get_auth_request con 0x7f818c06d730 auth_method 0 2024-05-04T17:44:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.253+0000 7f8187fff640 10 client.4243.objecter ms_handle_connect 0x7f818c06d730 2024-05-04T17:44:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.253+0000 7f819effd640 1 --2- 172.21.15.19:0/1290319004 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f818c06d730 0x7f818c06fbf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8190005880 tx=0x7f8190009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:44:57.280 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.280+0000 7f8187fff640 1 -- 172.21.15.19:0/1290319004 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f81940ab710 con 0x7f81a00a9ac0 2024-05-04T17:44:57.281 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.280+0000 7f8187fff640 10 client.4243.objecter ms_dispatch 0x7f81a0000b90 osd_map(10..10 src has 1..10) 2024-05-04T17:44:57.281 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.280+0000 7f8187fff640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-05-04T17:44:57.281 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.280+0000 7f8187fff640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-05-04T17:44:57.281 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.280+0000 7f8187fff640 20 client.4243.objecter dump_active .. 0 homeless 2024-05-04T17:44:57.282 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.281+0000 7f8187fff640 1 -- 172.21.15.19:0/1290319004 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8194031800 con 0x7f81a00a9ac0 2024-05-04T17:44:57.282 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.281+0000 7f8187fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:44:57.282 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.281+0000 7f8187fff640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:44:57.650 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.649+0000 7f81a5997640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:57.650 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.649+0000 7f81a5997640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:57.650 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.649+0000 7f81a5997640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:44:57.650 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.649+0000 7f81a5997640 1 -- 172.21.15.19:0/1290319004 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81a014d470 con 0x7f81a00a9ac0 2024-05-04T17:44:57.675 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.674+0000 7f8187fff640 1 -- 172.21.15.19:0/1290319004 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f819407b510 con 0x7f81a00a9ac0 2024-05-04T17:44:57.676 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.675+0000 7f8187fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:44:57.676 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.675+0000 7f8187fff640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:44:57.676 INFO:teuthology.misc.health.smithi019.stdout: 2024-05-04T17:44:57.676 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"18ced2df-a29f-46cc-b71a-1855bfb60c47","created":"2024-05-04T17:44:16.995866+0000","modified":"2024-05-04T17:44:23.464900+0000","last_up_change":"0.000000","last_in_change":"2024-05-04T17:44:23.464900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4b749ec0-22b6-458c-a7af-2ccace806ab7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3aa520fe-39c4-4fa2-860b-f4d0cc04ca97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16ed21c9-e5c3-4d65-b4d1-e4927a0fe919","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"45aa2c0f-cfae-478c-a60c-4c1491e45525","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb00f84f-d78c-461c-9843-73a9da4b80a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d07ad403-291f-4bde-a64a-5181fb79e544","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5110d1ad-5634-4c8e-9338-d6c4229b1d63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b307b905-5cc8-447b-8765-d888ed6f655b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:44:57.678 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 10 librados: watch_flush enter 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 10 librados: watch_flush exit 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 20 client.4243.objecter shutdown clearing up homeless session... 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 10 client.4243.objecter successfully canceled tick 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 1 -- 172.21.15.19:0/1290319004 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f818c06d730 msgr2=0x7f818c06fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 1 --2- 172.21.15.19:0/1290319004 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f818c06d730 0x7f818c06fbf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8190005880 tx=0x7f8190009040 comp rx=0 tx=0).stop 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 10 monclient: shutdown 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 1 -- 172.21.15.19:0/1290319004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a00a9ac0 msgr2=0x7f81a01472e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 1 --2- 172.21.15.19:0/1290319004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a00a9ac0 0x7f81a01472e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f819402bee0 tx=0x7f81940046c0 comp rx=0 tx=0).stop 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 1 -- 172.21.15.19:0/1290319004 shutdown_connections 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 1 --2- 172.21.15.19:0/1290319004 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f818c06d730 0x7f818c06fbf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 1 --2- 172.21.15.19:0/1290319004 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f81a01437e0 0x7f81a01457c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 1 --2- 172.21.15.19:0/1290319004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81a00a9ac0 0x7f81a01472e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:57.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.677+0000 7f81a5997640 1 --2- 172.21.15.19:0/1290319004 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f81a00a96e0 0x7f81a0144a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:44:57.680 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.678+0000 7f81a5997640 1 -- 172.21.15.19:0/1290319004 >> 172.21.15.19:0/1290319004 conn(0x7f81a00678d0 msgr2=0x7f81a013e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:44:57.680 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.679+0000 7f81a5997640 1 -- 172.21.15.19:0/1290319004 shutdown_connections 2024-05-04T17:44:57.680 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.679+0000 7f81a5997640 1 -- 172.21.15.19:0/1290319004 wait complete. 2024-05-04T17:44:57.680 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:44:57.679+0000 7f81a5997640 1 librados: shutdown 2024-05-04T17:44:57.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-04T17:44:58.042 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-05-04T17:44:58.038+0000 5dce500 -1 osd.5 0 log_to_monitors true 2024-05-04T17:44:58.423 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:44:58.414+0000 5dce500 -1 osd.6 0 log_to_monitors true 2024-05-04T17:44:58.474 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:44:58.469+0000 5dce500 -1 osd.4 0 log_to_monitors true 2024-05-04T17:44:58.559 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:44:58.555+0000 5dce500 -1 osd.7 0 log_to_monitors true 2024-05-04T17:44:59.090 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:44:59.083+0000 5dce500 -1 osd.2 0 log_to_monitors true 2024-05-04T17:45:00.303 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:00.299+0000 5dce500 -1 osd.0 0 log_to_monitors true 2024-05-04T17:45:00.344 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:45:00.340+0000 5dce500 -1 osd.3 0 log_to_monitors true 2024-05-04T17:45:00.568 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:45:00.564+0000 5dce500 -1 osd.1 0 log_to_monitors true 2024-05-04T17:45:01.216 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-05-04T17:45:01.211+0000 c49a640 -1 osd.5 0 waiting for initial osdmap 2024-05-04T17:45:01.577 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-05-04T17:45:01.562+0000 10260640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:45:02.296 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:45:02.289+0000 c49a640 -1 osd.7 0 waiting for initial osdmap 2024-05-04T17:45:02.324 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:45:02.320+0000 c49a640 -1 osd.2 0 waiting for initial osdmap 2024-05-04T17:45:02.677 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:45:02.661+0000 10260640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:45:02.684 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:45:02.669+0000 10260640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:45:03.365 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:45:03.358+0000 ac84640 -1 osd.3 0 waiting for initial osdmap 2024-05-04T17:45:03.839 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:45:03.825+0000 10260640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:45:04.499 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-04T17:45:04.594 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.593+0000 7f35d1179640 10 monclient: get_monmap_and_config 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.593+0000 7f35d1179640 10 monclient: build_initial_monmap 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.593+0000 7f35d1179640 10 monclient: monmap: 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:45:04.594747+0000 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:45:04.594747+0000 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:04.595 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:04.598 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.594+0000 7f35d1179640 1 Processor -- start 2024-05-04T17:45:04.598 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.594+0000 7f35d1179640 1 -- start start 2024-05-04T17:45:04.598 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.594+0000 7f35d1179640 10 monclient: init 2024-05-04T17:45:04.598 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:04.598 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-05-04T17:45:04.598 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f35cc0a96e0 0x7f35cc1430d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:04.598 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 10 monclient(hunting): picked mon.noname-a con 0x7f35cc0a96e0 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:04.598 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35cc0a9ac0 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:04.598 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f35cc143610 0x7f35cc146840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 10 monclient(hunting): picked mon.noname-c con 0x7f35cc143610 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35cc0abbf0 con 0x7f35cc0a9ac0 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f35cc0ab9a0 con 0x7f35cc0a96e0 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f35cc142bb0 con 0x7f35cc143610 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35d1179640 10 monclient(hunting): authenticate will time out at 1326.708944s 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.595+0000 7f35cad9d640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f35cc0a96e0 0x7f35cc1430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35ca59c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f35cc0a9ac0 0x7f35cc146130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:38008/0 (socket says 172.21.15.19:38008) 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35ca59c640 1 -- 172.21.15.19:0/4084778121 learned_addr learned my addr 172.21.15.19:0/4084778121 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35cb59e640 1 --2- 172.21.15.19:0/4084778121 >> v2:172.21.15.186:3300/0 conn(0x7f35cc143610 0x7f35cc146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:04.599 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35cad9d640 10 monclient(hunting): get_auth_request con 0x7f35cc0a96e0 auth_method 0 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35cad9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35cb59e640 10 monclient(hunting): get_auth_request con 0x7f35cc143610 auth_method 0 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35cb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35cb59e640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35cb59e640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.596+0000 7f35cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.597+0000 7f35cb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.597+0000 7f35cb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:04.600 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.597+0000 7f35cb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:04.601 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.598+0000 7f35cad9d640 10 monclient(hunting): handle_auth_done global_id 4233 payload 274 2024-05-04T17:45:04.601 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.598+0000 7f35cad9d640 1 -- 172.21.15.19:0/4084778121 >> v2:172.21.15.186:3300/0 conn(0x7f35cc143610 msgr2=0x7f35cc146840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:04.601 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.598+0000 7f35cad9d640 1 --2- 172.21.15.19:0/4084778121 >> v2:172.21.15.186:3300/0 conn(0x7f35cc143610 0x7f35cc146840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:04.601 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.598+0000 7f35cad9d640 1 -- 172.21.15.19:0/4084778121 >> v1:172.21.15.186:6789/0 conn(0x7f35cc0a9ac0 legacy=0x7f35cc146130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:04.601 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.598+0000 7f35cad9d640 10 monclient: _finish_hunting 0 2024-05-04T17:45:04.602 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.598+0000 7f35cad9d640 1 monclient: found mon.noname-a 2024-05-04T17:45:04.602 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.598+0000 7f35cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:04.602 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.598+0000 7f35cad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:04.602 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.598+0000 7f35cad9d640 1 -- 172.21.15.19:0/4084778121 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35cc143aa0 con 0x7f35cc0a96e0 2024-05-04T17:45:04.602 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.600+0000 7f35cad9d640 1 --2- 172.21.15.19:0/4084778121 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35cc0a96e0 0x7f35cc1430d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0ac0c0 tx=0x7f35bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fae9e69038c3656 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:04.606 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 1 -- 172.21.15.19:0/4084778121 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc035020 con 0x7f35cc0a96e0 2024-05-04T17:45:04.606 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:04.606 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:04.606 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 10 monclient: dump: 2024-05-04T17:45:04.606 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:45:04.606 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:04.606 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:04.606 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 10 monclient: _finish_auth 0 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 10 monclient: _check_auth_tickets 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 1 -- 172.21.15.19:0/4084778121 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35bc031830 con 0x7f35cc0a96e0 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35d1179640 5 monclient: authenticate success, global_id 4233 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35d1179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35d1179640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:04.607 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35d1179640 4 set_mon_vals no callback set 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 1 -- 172.21.15.19:0/4084778121 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc031c00 con 0x7f35cc0a96e0 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.605+0000 7f35c9d9b640 10 monclient: dump: 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:04.608 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:04.611 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.606+0000 7f35d1179640 10 monclient: shutdown 2024-05-04T17:45:04.611 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.606+0000 7f35d1179640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:04.611 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.606+0000 7f35d1179640 1 -- 172.21.15.19:0/4084778121 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35cc0a96e0 msgr2=0x7f35cc1430d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.606+0000 7f35d1179640 1 --2- 172.21.15.19:0/4084778121 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35cc0a96e0 0x7f35cc1430d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0ac0c0 tx=0x7f35bc02b8d0 comp rx=0 tx=0).stop 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.606+0000 7f35d1179640 1 -- 172.21.15.19:0/4084778121 shutdown_connections 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.606+0000 7f35d1179640 1 --2- 172.21.15.19:0/4084778121 >> v2:172.21.15.186:3300/0 conn(0x7f35cc143610 0x7f35cc146840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.606+0000 7f35d1179640 1 --2- 172.21.15.19:0/4084778121 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35cc0a96e0 0x7f35cc1430d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.606+0000 7f35d1179640 1 -- 172.21.15.19:0/4084778121 >> 172.21.15.19:0/4084778121 conn(0x7f35cc0678d0 msgr2=0x7f35cc0b4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.606+0000 7f35d1179640 1 -- 172.21.15.19:0/4084778121 shutdown_connections 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.606+0000 7f35d1179640 1 -- 172.21.15.19:0/4084778121 wait complete. 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.607+0000 7f35d1179640 10 monclient: build_initial_monmap 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.607+0000 7f35d1179640 10 monclient: monmap: 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-05-04T17:45:04.612 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.608+0000 7f35d1179640 1 librados: starting msgr at 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.608+0000 7f35d1179640 1 librados: starting objecter 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.608+0000 7f35d1179640 1 Processor -- start 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.608+0000 7f35d1179640 1 -- start start 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.608+0000 7f35d1179640 1 librados: setting wanted keys 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.608+0000 7f35d1179640 1 librados: calling monclient init 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.608+0000 7f35d1179640 10 monclient: init 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T17:45:04.613 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc0a96e0 0x7f35cc1cf660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 10 monclient(hunting): picked mon.noname-b con 0x7f35cc0a96e0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35cc0a9ac0 0x7f35cc1ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 10 monclient(hunting): picked mon.noname-a con 0x7f35cc0a9ac0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f35cc143610 0x7f35cc1cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 10 monclient(hunting): picked mon.noname-c con 0x7f35cc143610 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc0aaaa0 con 0x7f35cc0a9ac0 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc0aa920 con 0x7f35cc0a96e0 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f35cc146d80 con 0x7f35cc143610 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:04.614 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.609+0000 7f35d1179640 10 monclient(hunting): authenticate will time out at 1326.723118s 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cad9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc0a96e0 0x7f35cc1cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cb59e640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f35cc143610 0x7f35cc1cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cad9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc0a96e0 0x7f35cc1cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:55226/0 (socket says 172.21.15.19:55226) 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cad9d640 1 -- 172.21.15.19:0/1794807881 learned_addr learned my addr 172.21.15.19:0/1794807881 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cad9d640 10 monclient(hunting): get_auth_request con 0x7f35cc0a96e0 auth_method 0 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cad9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cb59e640 10 monclient(hunting): get_auth_request con 0x7f35cc143610 auth_method 0 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cb59e640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cb59e640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:04.615 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.610+0000 7f35cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:04.617 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.611+0000 7f35cb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:04.617 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.611+0000 7f35cb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:04.617 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.611+0000 7f35cb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:04.617 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.613+0000 7f35ca59c640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35cc0a9ac0 0x7f35cc1ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:04.617 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.613+0000 7f35cad9d640 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2024-05-04T17:45:04.617 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.613+0000 7f35cad9d640 1 -- 172.21.15.19:0/1794807881 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f35cc143610 msgr2=0x7f35cc1cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:04.617 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.613+0000 7f35cad9d640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f35cc143610 0x7f35cc1cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:04.617 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.613+0000 7f35cad9d640 1 -- 172.21.15.19:0/1794807881 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35cc0a9ac0 msgr2=0x7f35cc1ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:04.617 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.613+0000 7f35cad9d640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35cc0a9ac0 0x7f35cc1ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:04.617 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.613+0000 7f35cad9d640 10 monclient: _finish_hunting 0 2024-05-04T17:45:04.617 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.613+0000 7f35cad9d640 1 monclient: found mon.noname-b 2024-05-04T17:45:04.618 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.613+0000 7f35cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:04.618 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.613+0000 7f35cad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:04.618 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.613+0000 7f35cad9d640 1 -- 172.21.15.19:0/1794807881 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35cc1ce910 con 0x7f35cc0a96e0 2024-05-04T17:45:04.618 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.615+0000 7f35cad9d640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc0a96e0 0x7f35cc1cf660 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f35bc02bee0 tx=0x7f35bc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:04.618 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.615+0000 7f35b37fe640 10 client.?.objecter ms_handle_connect 0x7f35cc0a96e0 2024-05-04T17:45:04.618 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.615+0000 7f35b37fe640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:04.621 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 1 -- 172.21.15.19:0/1794807881 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc041070 con 0x7f35cc0a96e0 2024-05-04T17:45:04.621 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:04.621 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 10 monclient: dump: 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:04.622 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 1 -- 172.21.15.19:0/1794807881 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35bc03f070 con 0x7f35cc0a96e0 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35d1179640 5 monclient: authenticate success, global_id 4276 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 1 -- 172.21.15.19:0/1794807881 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc035040 con 0x7f35cc0a96e0 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35d1179640 10 monclient: _renew_subs 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35d1179640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35d1179640 1 -- 172.21.15.19:0/1794807881 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35cc1cebf0 con 0x7f35cc0a96e0 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35c959a640 4 set_mon_vals no callback set 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35b37fe640 10 monclient: dump: 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:45:04.623 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35d1179640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35d1179640 10 monclient: _renew_subs 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35d1179640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35d1179640 1 -- 172.21.15.19:0/1794807881 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35cc1cfba0 con 0x7f35cc0a96e0 2024-05-04T17:45:04.624 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.620+0000 7f35d1179640 1 librados: init done 2024-05-04T17:45:04.625 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.621+0000 7f35d1179640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:04.625 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.621+0000 7f35d1179640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:04.625 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.621+0000 7f35d1179640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:04.625 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.621+0000 7f35d1179640 1 -- 172.21.15.19:0/1794807881 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35cc04f5f0 con 0x7f35cc0a96e0 2024-05-04T17:45:04.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.678+0000 7f35b37fe640 1 -- 172.21.15.19:0/1794807881 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f35bc038020 con 0x7f35cc0a96e0 2024-05-04T17:45:04.679 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.679+0000 7f35b37fe640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f35b806d780 0x7f35b806fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:04.680 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.679+0000 7f35ca59c640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f35b806d780 0x7f35b806fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:04.680 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.679+0000 7f35ca59c640 10 monclient: get_auth_request con 0x7f35b806d780 auth_method 0 2024-05-04T17:45:04.680 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.680+0000 7f35ca59c640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f35b806d780 0x7f35b806fc40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f35c0007fd0 tx=0x7f35c0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:04.680 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.680+0000 7f35b37fe640 10 client.4276.objecter ms_handle_connect 0x7f35b806d780 2024-05-04T17:45:04.890 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.889+0000 7f35b37fe640 1 -- 172.21.15.19:0/1794807881 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3446+0+0 (secure 0 0 0) 0x7f35bc0789e0 con 0x7f35cc0a96e0 2024-05-04T17:45:04.890 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.889+0000 7f35b37fe640 10 client.4276.objecter ms_dispatch 0x7f35cc000b90 osd_map(15..15 src has 1..15) 2024-05-04T17:45:04.890 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.889+0000 7f35b37fe640 3 client.4276.objecter handle_osd_map got epochs [15,15] > 0 2024-05-04T17:45:04.890 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.889+0000 7f35b37fe640 3 client.4276.objecter handle_osd_map decoding full epoch 15 2024-05-04T17:45:04.890 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.889+0000 7f35b37fe640 20 client.4276.objecter dump_active .. 0 homeless 2024-05-04T17:45:04.900 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.899+0000 7f35b37fe640 1 -- 172.21.15.19:0/1794807881 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f35bc07b040 con 0x7f35cc0a96e0 2024-05-04T17:45:04.900 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.899+0000 7f35b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:04.901 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:04.899+0000 7f35b37fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:05.098 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.097+0000 7f35d1179640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:05.098 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.097+0000 7f35d1179640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:05.098 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.097+0000 7f35d1179640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:05.098 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.097+0000 7f35d1179640 1 -- 172.21.15.19:0/1794807881 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f35cc1cc870 con 0x7f35cc0a96e0 2024-05-04T17:45:05.239 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.239+0000 7f35b37fe640 1 -- 172.21.15.19:0/1794807881 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+10920 (secure 0 0 0) 0x7f35bc031800 con 0x7f35cc0a96e0 2024-05-04T17:45:05.240 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.239+0000 7f35b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:05.240 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.239+0000 7f35b37fe640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:05.240 INFO:teuthology.misc.health.smithi019.stdout: 2024-05-04T17:45:05.240 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":16,"fsid":"18ced2df-a29f-46cc-b71a-1855bfb60c47","created":"2024-05-04T17:44:16.995866+0000","modified":"2024-05-04T17:45:05.022474+0000","last_up_change":"2024-05-04T17:45:05.022474+0000","last_in_change":"2024-05-04T17:44:23.464900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T17:45:03.247977+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b749ec0-22b6-458c-a7af-2ccace806ab7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3aa520fe-39c4-4fa2-860b-f4d0cc04ca97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16ed21c9-e5c3-4d65-b4d1-e4927a0fe919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6817","nonce":3744614279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6820","nonce":3744614279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6825","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6827","nonce":3744614279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6823","nonce":3744614279}]},"public_addr":"172.21.15.19:6817/3744614279","cluster_addr":"172.21.15.19:6820/3744614279","heartbeat_back_addr":"172.21.15.19:6827/3744614279","heartbeat_front_addr":"172.21.15.19:6823/3744614279","state":["exists","up"]},{"osd":3,"uuid":"45aa2c0f-cfae-478c-a60c-4c1491e45525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6819","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6821","nonce":2537429684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6826","nonce":2537429684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6831","nonce":2537429684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6829","nonce":2537429684}]},"public_addr":"172.21.15.19:6821/2537429684","cluster_addr":"172.21.15.19:6826/2537429684","heartbeat_back_addr":"172.21.15.19:6831/2537429684","heartbeat_front_addr":"172.21.15.19:6829/2537429684","state":["exists","up"]},{"osd":4,"uuid":"cb00f84f-d78c-461c-9843-73a9da4b80a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d07ad403-291f-4bde-a64a-5181fb79e544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6811","nonce":3728813703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6813","nonce":3728813703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6817","nonce":3728813703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6815","nonce":3728813703}]},"public_addr":"172.21.15.186:6811/3728813703","cluster_addr":"172.21.15.186:6813/3728813703","heartbeat_back_addr":"172.21.15.186:6817/3728813703","heartbeat_front_addr":"172.21.15.186:6815/3728813703","state":["exists","up"]},{"osd":6,"uuid":"5110d1ad-5634-4c8e-9338-d6c4229b1d63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b307b905-5cc8-447b-8765-d888ed6f655b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6827","nonce":1403900459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6829","nonce":1403900459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6833","nonce":1403900459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6831","nonce":1403900459}]},"public_addr":"172.21.15.186:6827/1403900459","cluster_addr":"172.21.15.186:6829/1403900459","heartbeat_back_addr":"172.21.15.186:6833/1403900459","heartbeat_front_addr":"172.21.15.186:6831/1403900459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:01.075527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:44:59.984305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.439608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:45:05.242 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.241+0000 7f35d1179640 10 librados: watch_flush enter 2024-05-04T17:45:05.243 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.241+0000 7f35d1179640 10 librados: watch_flush exit 2024-05-04T17:45:05.243 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.241+0000 7f35d1179640 20 client.4276.objecter shutdown clearing up homeless session... 2024-05-04T17:45:05.243 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.241+0000 7f35d1179640 10 client.4276.objecter successfully canceled tick 2024-05-04T17:45:05.243 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.242+0000 7f35d1179640 1 -- 172.21.15.19:0/1794807881 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f35b806d780 msgr2=0x7f35b806fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:05.243 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.242+0000 7f35d1179640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f35b806d780 0x7f35b806fc40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f35c0007fd0 tx=0x7f35c0009040 comp rx=0 tx=0).stop 2024-05-04T17:45:05.243 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.242+0000 7f35d1179640 10 monclient: shutdown 2024-05-04T17:45:05.243 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.242+0000 7f35d1179640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:05.243 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.242+0000 7f35d1179640 1 -- 172.21.15.19:0/1794807881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc0a96e0 msgr2=0x7f35cc1cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:05.243 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.242+0000 7f35d1179640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc0a96e0 0x7f35cc1cf660 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f35bc02bee0 tx=0x7f35bc0046c0 comp rx=0 tx=0).stop 2024-05-04T17:45:05.245 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.244+0000 7f35d1179640 1 -- 172.21.15.19:0/1794807881 shutdown_connections 2024-05-04T17:45:05.245 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.244+0000 7f35d1179640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f35b806d780 0x7f35b806fc40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:05.245 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.244+0000 7f35d1179640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f35cc143610 0x7f35cc1cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:05.245 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.244+0000 7f35d1179640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f35cc0a9ac0 0x7f35cc1ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:05.245 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.244+0000 7f35d1179640 1 --2- 172.21.15.19:0/1794807881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35cc0a96e0 0x7f35cc1cf660 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:05.245 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.244+0000 7f35d1179640 1 -- 172.21.15.19:0/1794807881 >> 172.21.15.19:0/1794807881 conn(0x7f35cc0678d0 msgr2=0x7f35cc13f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:05.245 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.244+0000 7f35d1179640 1 -- 172.21.15.19:0/1794807881 shutdown_connections 2024-05-04T17:45:05.245 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.244+0000 7f35d1179640 1 -- 172.21.15.19:0/1794807881 wait complete. 2024-05-04T17:45:05.245 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:05.244+0000 7f35d1179640 1 librados: shutdown 2024-05-04T17:45:05.258 DEBUG:teuthology.misc:4 of 8 OSDs are up 2024-05-04T17:45:09.267 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:45:09.223+0000 5dce500 -1 bdev(0x18ada760 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-05-04T17:45:09.275 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:45:09.219+0000 5dce500 -1 bdev(0x18ab28f0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-04T17:45:09.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:45:09.280+0000 5dce500 -1 bdev(0x18ada760 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-05-04T17:45:09.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:45:09.275+0000 5dce500 -1 bdev(0x18ab28f0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-05-04T17:45:09.483 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:45:09.326+0000 5dce500 -1 bdev(0x18ab28f0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-04T17:45:09.489 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:45:09.486+0000 5dce500 -1 bdev(0x18ab28f0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-05-04T17:45:09.680 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:45:09.480+0000 5dce500 -1 bdev(0x18ada760 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-05-04T17:45:09.695 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:45:09.679+0000 5dce500 -1 bdev(0x18ada760 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-05-04T17:45:09.900 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:45:09.657+0000 5dce500 -1 bdev(0x18ab28f0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-04T17:45:09.924 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:45:09.900+0000 5dce500 -1 bdev(0x18ab28f0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-05-04T17:45:10.048 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:45:09.927+0000 5dce500 -1 bdev(0x18ab28f0 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 2 2024-05-04T17:45:10.076 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:45:09.697+0000 5dce500 -1 bdev(0x18ada760 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-05-04T17:45:10.175 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:45:10.097+0000 5dce500 -1 bdev(0x18ada760 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-05-04T17:45:11.087 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:45:11.083+0000 c49a640 -1 osd.6 0 waiting for initial osdmap 2024-05-04T17:45:11.166 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:45:11.160+0000 c49a640 -1 osd.4 0 waiting for initial osdmap 2024-05-04T17:45:11.619 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:11.313+0000 5dce500 -1 bdev(0x18b009e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 2 2024-05-04T17:45:11.697 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:45:11.681+0000 14268640 -1 osd.6 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:45:11.708 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:45:11.693+0000 14268640 -1 osd.4 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:45:11.741 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:11.616+0000 5dce500 -1 bdev(0x18b009e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-04T17:45:11.768 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:11.636+0000 5dce500 -1 bdev(0x18b009e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-04T17:45:11.795 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:11.705+0000 5dce500 -1 bdev(0x18b009e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-04T17:45:11.950 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:11.741+0000 5dce500 -1 bdev(0x18b009e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-04T17:45:11.951 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:11.795+0000 5dce500 -1 bdev(0x18b009e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-04T17:45:11.995 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:11.950+0000 5dce500 -1 bdev(0x18b009e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-04T17:45:12.063 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-04T17:45:12.077 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:12.076+0000 5dce500 -1 bdev(0x18b009e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 2 2024-05-04T17:45:12.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.160+0000 7f8ec72a7640 10 monclient: get_monmap_and_config 2024-05-04T17:45:12.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.160+0000 7f8ec72a7640 10 monclient: build_initial_monmap 2024-05-04T17:45:12.161 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.160+0000 7f8ec72a7640 10 monclient: monmap: 2024-05-04T17:45:12.161 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:45:12.161 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:12.161 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:45:12.160977+0000 2024-05-04T17:45:12.161 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:45:12.160977+0000 2024-05-04T17:45:12.161 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:12.161 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:12.161 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:12.161 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:12.162 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:12.162 INFO:teuthology.misc.health.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:12.162 INFO:teuthology.misc.health.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:12.162 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.160+0000 7f8ec72a7640 1 Processor -- start 2024-05-04T17:45:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 1 -- start start 2024-05-04T17:45:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 10 monclient: init 2024-05-04T17:45:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-05-04T17:45:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f8ec00a96e0 0x7f8ec01430d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 10 monclient(hunting): picked mon.noname-a con 0x7f8ec00a96e0 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ec00a9ac0 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8ec0143610 0x7f8ec0146840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 10 monclient(hunting): picked mon.noname-c con 0x7f8ec0143610 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ec00abbf0 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.164 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f8ec00ab9a0 con 0x7f8ec00a96e0 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8ec0142bb0 con 0x7f8ec0143610 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.161+0000 7f8ec72a7640 10 monclient(hunting): authenticate will time out at 1334.275445s 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.162+0000 7f8ec5844640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8ec0143610 0x7f8ec0146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.162+0000 7f8ec5844640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8ec0143610 0x7f8ec0146840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:55018/0 (socket says 172.21.15.19:55018) 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.162+0000 7f8ec5844640 1 -- 172.21.15.19:0/3292602155 learned_addr learned my addr 172.21.15.19:0/3292602155 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.162+0000 7f8ec5844640 10 monclient(hunting): get_auth_request con 0x7f8ec0143610 auth_method 0 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.162+0000 7f8ec5844640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.162+0000 7f8ec5844640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.162+0000 7f8ec5844640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:12.165 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.163+0000 7f8ec5844640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:12.166 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.163+0000 7f8ec5844640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:12.166 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.163+0000 7f8ec5844640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:12.166 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.163+0000 7f8ec5043640 1 --2- 172.21.15.19:0/3292602155 >> v2:172.21.15.19:3300/0 conn(0x7f8ec00a96e0 0x7f8ec01430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:12.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.164+0000 7f8ec5043640 10 monclient(hunting): get_auth_request con 0x7f8ec00a96e0 auth_method 0 2024-05-04T17:45:12.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.164+0000 7f8ec5043640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:12.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.164+0000 7f8ec5043640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:12.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.164+0000 7f8ec5043640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:12.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.165+0000 7f8ec5844640 10 monclient(hunting): handle_auth_done global_id 4285 payload 274 2024-05-04T17:45:12.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.165+0000 7f8ec5844640 1 -- 172.21.15.19:0/3292602155 >> v2:172.21.15.19:3300/0 conn(0x7f8ec00a96e0 msgr2=0x7f8ec01430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:12.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.165+0000 7f8ec5844640 1 --2- 172.21.15.19:0/3292602155 >> v2:172.21.15.19:3300/0 conn(0x7f8ec00a96e0 0x7f8ec01430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:12.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.165+0000 7f8ec5844640 1 -- 172.21.15.19:0/3292602155 >> v1:172.21.15.19:6789/0 conn(0x7f8ec00a9ac0 legacy=0x7f8ec0146130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:12.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.165+0000 7f8ec5844640 10 monclient: _finish_hunting 0 2024-05-04T17:45:12.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.165+0000 7f8ec5844640 1 monclient: found mon.noname-c 2024-05-04T17:45:12.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.165+0000 7f8ec5844640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:12.167 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.165+0000 7f8ec5844640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:12.168 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.165+0000 7f8ec5844640 1 -- 172.21.15.19:0/3292602155 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec0143aa0 con 0x7f8ec0143610 2024-05-04T17:45:12.168 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.167+0000 7f8ec5844640 1 --2- 172.21.15.19:0/3292602155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0143610 0x7f8ec0146840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00ac0c0 tx=0x7f8eb802bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d96db0a77b1e366f server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:12.170 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:12.163+0000 5dce500 -1 bdev(0x18b009e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 2 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 1 -- 172.21.15.19:0/3292602155 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb802cb10 con 0x7f8ec0143610 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 10 monclient: dump: 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:12.172 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 10 monclient: _finish_auth 0 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 10 monclient: _check_auth_tickets 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 1 -- 172.21.15.19:0/3292602155 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8eb802ccb0 con 0x7f8ec0143610 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8ec72a7640 5 monclient: authenticate success, global_id 4285 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8ec72a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8ec72a7640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8ec72a7640 4 set_mon_vals no callback set 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 1 -- 172.21.15.19:0/3292602155 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb802ce60 con 0x7f8ec0143610 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8eb7fff640 10 monclient: dump: 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:12.173 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:12.174 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:12.174 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:12.174 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:12.174 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:12.174 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:12.174 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:12.174 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:12.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8ec72a7640 10 monclient: shutdown 2024-05-04T17:45:12.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8ec72a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:12.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8ec72a7640 1 -- 172.21.15.19:0/3292602155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0143610 msgr2=0x7f8ec0146840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:12.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.171+0000 7f8ec72a7640 1 --2- 172.21.15.19:0/3292602155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0143610 0x7f8ec0146840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00ac0c0 tx=0x7f8eb802bf70 comp rx=0 tx=0).stop 2024-05-04T17:45:12.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.172+0000 7f8ec72a7640 1 -- 172.21.15.19:0/3292602155 shutdown_connections 2024-05-04T17:45:12.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.172+0000 7f8ec72a7640 1 --2- 172.21.15.19:0/3292602155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec0143610 0x7f8ec0146840 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:12.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.172+0000 7f8ec72a7640 1 --2- 172.21.15.19:0/3292602155 >> v2:172.21.15.19:3300/0 conn(0x7f8ec00a96e0 0x7f8ec01430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:12.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.172+0000 7f8ec72a7640 1 -- 172.21.15.19:0/3292602155 >> 172.21.15.19:0/3292602155 conn(0x7f8ec00678d0 msgr2=0x7f8ec00b4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:12.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.172+0000 7f8ec72a7640 1 -- 172.21.15.19:0/3292602155 shutdown_connections 2024-05-04T17:45:12.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.172+0000 7f8ec72a7640 1 -- 172.21.15.19:0/3292602155 wait complete. 2024-05-04T17:45:12.178 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.172+0000 7f8ec72a7640 10 monclient: build_initial_monmap 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.172+0000 7f8ec72a7640 10 monclient: monmap: 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 1 librados: starting msgr at 2024-05-04T17:45:12.179 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 1 librados: starting objecter 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 1 Processor -- start 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 1 -- start start 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 1 librados: setting wanted keys 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 1 librados: calling monclient init 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 10 monclient: init 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8ec00a96e0 0x7f8ec01db4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 10 monclient(hunting): picked mon.noname-c con 0x7f8ec00a96e0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec00a9ac0 0x7f8ec0144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 10 monclient(hunting): picked mon.noname-b con 0x7f8ec00a9ac0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8ec0143610 0x7f8ec01446d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:12.180 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 10 monclient(hunting): picked mon.noname-a con 0x7f8ec0143610 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec00aaaa0 con 0x7f8ec0143610 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8ec00aa920 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f8ec0146d80 con 0x7f8ec00a96e0 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.173+0000 7f8ec72a7640 10 monclient(hunting): authenticate will time out at 1334.287611s 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.174+0000 7f8ec4842640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec00a9ac0 0x7f8ec0144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.174+0000 7f8ec5043640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8ec00a96e0 0x7f8ec01db4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.174+0000 7f8ec5844640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8ec0143610 0x7f8ec01446d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.174+0000 7f8ec4842640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec00a9ac0 0x7f8ec0144190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:55020/0 (socket says 172.21.15.19:55020) 2024-05-04T17:45:12.181 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.174+0000 7f8ec4842640 1 -- 172.21.15.19:0/2764529440 learned_addr learned my addr 172.21.15.19:0/2764529440 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.174+0000 7f8ec4842640 10 monclient(hunting): get_auth_request con 0x7f8ec00a9ac0 auth_method 0 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.174+0000 7f8ec4842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.174+0000 7f8ec4842640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.174+0000 7f8ec4842640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5844640 10 monclient(hunting): get_auth_request con 0x7f8ec0143610 auth_method 0 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5844640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5844640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5844640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5043640 10 monclient(hunting): get_auth_request con 0x7f8ec00a96e0 auth_method 0 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5043640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5043640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5043640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec4842640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:12.182 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec4842640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:12.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec4842640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:12.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5043640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:12.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5043640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:12.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5043640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:12.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5844640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:12.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5844640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:12.183 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.175+0000 7f8ec5844640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:12.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.178+0000 7f8ec4842640 10 monclient(hunting): handle_auth_done global_id 4288 payload 824 2024-05-04T17:45:12.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.178+0000 7f8ec4842640 1 -- 172.21.15.19:0/2764529440 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8ec00a96e0 msgr2=0x7f8ec01db4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:12.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.178+0000 7f8ec4842640 1 --2- 172.21.15.19:0/2764529440 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8ec00a96e0 0x7f8ec01db4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:12.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.178+0000 7f8ec4842640 1 -- 172.21.15.19:0/2764529440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8ec0143610 msgr2=0x7f8ec01446d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:12.184 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.178+0000 7f8ec4842640 1 --2- 172.21.15.19:0/2764529440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8ec0143610 0x7f8ec01446d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:12.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.178+0000 7f8ec4842640 10 monclient: _finish_hunting 0 2024-05-04T17:45:12.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.178+0000 7f8ec4842640 1 monclient: found mon.noname-b 2024-05-04T17:45:12.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.178+0000 7f8ec4842640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:12.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.178+0000 7f8ec4842640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:12.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.178+0000 7f8ec4842640 1 -- 172.21.15.19:0/2764529440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ec00ae8b0 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.180+0000 7f8ec4842640 1 --2- 172.21.15.19:0/2764529440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec00a9ac0 0x7f8ec0144190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000c430 tx=0x7f8eb000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:12.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.180+0000 7f8eb5ffb640 10 client.?.objecter ms_handle_connect 0x7f8ec00a9ac0 2024-05-04T17:45:12.185 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.180+0000 7f8eb5ffb640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 1 -- 172.21.15.19:0/2764529440 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb00052d0 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 10 monclient: dump: 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:12.187 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 10 monclient: _finish_auth 0 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 10 monclient: _check_auth_tickets 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 1 -- 172.21.15.19:0/2764529440 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8eb0005490 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 1 -- 172.21.15.19:0/2764529440 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0007260 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb5ffb640 10 monclient: dump: 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:12.188 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8eb77fe640 4 set_mon_vals no callback set 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8ec72a7640 5 monclient: authenticate success, global_id 4288 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8ec72a7640 10 monclient: _renew_subs 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8ec72a7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8ec72a7640 1 -- 172.21.15.19:0/2764529440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ec00aeb90 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8ec72a7640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8ec72a7640 10 monclient: _renew_subs 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8ec72a7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8ec72a7640 1 -- 172.21.15.19:0/2764529440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ec01460c0 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.189 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.185+0000 7f8ec72a7640 1 librados: init done 2024-05-04T17:45:12.190 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.186+0000 7f8ec72a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:12.190 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.186+0000 7f8ec72a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:12.190 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.186+0000 7f8ec72a7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:12.190 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.186+0000 7f8ec72a7640 1 -- 172.21.15.19:0/2764529440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ec004f5f0 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.206 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.205+0000 7f8eb5ffb640 1 -- 172.21.15.19:0/2764529440 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8eb0007400 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.206 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.206+0000 7f8eb5ffb640 1 --2- 172.21.15.19:0/2764529440 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f8ea406d810 0x7f8ea406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:12.207 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.206+0000 7f8ec5043640 1 --2- 172.21.15.19:0/2764529440 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f8ea406d810 0x7f8ea406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:12.207 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.206+0000 7f8ec5043640 10 monclient: get_auth_request con 0x7f8ea406d810 auth_method 0 2024-05-04T17:45:12.208 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.207+0000 7f8ec5043640 1 --2- 172.21.15.19:0/2764529440 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f8ea406d810 0x7f8ea406fcd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc006fd0 tx=0x7f8ebc008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:12.208 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.207+0000 7f8eb5ffb640 10 client.4288.objecter ms_handle_connect 0x7f8ea406d810 2024-05-04T17:45:12.323 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:12.210+0000 5dce500 -1 bdev(0x18b009e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 3 2024-05-04T17:45:12.398 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:12.391+0000 5dce500 -1 bdev(0x18b009e0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-05-04T17:45:12.453 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.452+0000 7f8eb5ffb640 1 -- 172.21.15.19:0/2764529440 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3737+0+0 (secure 0 0 0) 0x7f8eb0056be0 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.454 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.452+0000 7f8eb5ffb640 10 client.4288.objecter ms_dispatch 0x7f8ec0000b90 osd_map(18..18 src has 1..18) 2024-05-04T17:45:12.454 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.452+0000 7f8eb5ffb640 3 client.4288.objecter handle_osd_map got epochs [18,18] > 0 2024-05-04T17:45:12.454 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.452+0000 7f8eb5ffb640 3 client.4288.objecter handle_osd_map decoding full epoch 18 2024-05-04T17:45:12.454 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.452+0000 7f8eb5ffb640 20 client.4288.objecter dump_active .. 0 homeless 2024-05-04T17:45:12.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.463+0000 7f8eb5ffb640 1 -- 172.21.15.19:0/2764529440 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8eb0058b40 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.463+0000 7f8eb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:12.464 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.463+0000 7f8eb5ffb640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:12.722 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.721+0000 7f8ec72a7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:12.722 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.721+0000 7f8ec72a7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:12.722 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.721+0000 7f8ec72a7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:12.722 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:12.721+0000 7f8ec72a7640 1 -- 172.21.15.19:0/2764529440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ec0073100 con 0x7f8ec00a9ac0 2024-05-04T17:45:12.904 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:45:12.682+0000 5dce500 -1 bdev(0x18e7e300 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 2 2024-05-04T17:45:12.929 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:45:12.721+0000 5dce500 -1 bdev(0x18e7e300 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-04T17:45:13.018 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.017+0000 7f8eb5ffb640 1 -- 172.21.15.19:0/2764529440 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+11891 (secure 0 0 0) 0x7f8eb005ce40 con 0x7f8ec00a9ac0 2024-05-04T17:45:13.018 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.017+0000 7f8eb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:13.018 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.017+0000 7f8eb5ffb640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:13.019 INFO:teuthology.misc.health.smithi019.stdout: 2024-05-04T17:45:13.019 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":19,"fsid":"18ced2df-a29f-46cc-b71a-1855bfb60c47","created":"2024-05-04T17:44:16.995866+0000","modified":"2024-05-04T17:45:12.723555+0000","last_up_change":"2024-05-04T17:45:12.723555+0000","last_in_change":"2024-05-04T17:44:23.464900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T17:45:03.247977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b749ec0-22b6-458c-a7af-2ccace806ab7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3aa520fe-39c4-4fa2-860b-f4d0cc04ca97","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"16ed21c9-e5c3-4d65-b4d1-e4927a0fe919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6817","nonce":3744614279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6820","nonce":3744614279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6825","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6827","nonce":3744614279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6823","nonce":3744614279}]},"public_addr":"172.21.15.19:6817/3744614279","cluster_addr":"172.21.15.19:6820/3744614279","heartbeat_back_addr":"172.21.15.19:6827/3744614279","heartbeat_front_addr":"172.21.15.19:6823/3744614279","state":["exists","up"]},{"osd":3,"uuid":"45aa2c0f-cfae-478c-a60c-4c1491e45525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6819","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6821","nonce":2537429684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6826","nonce":2537429684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6831","nonce":2537429684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6829","nonce":2537429684}]},"public_addr":"172.21.15.19:6821/2537429684","cluster_addr":"172.21.15.19:6826/2537429684","heartbeat_back_addr":"172.21.15.19:6831/2537429684","heartbeat_front_addr":"172.21.15.19:6829/2537429684","state":["exists","up"]},{"osd":4,"uuid":"cb00f84f-d78c-461c-9843-73a9da4b80a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6819","nonce":356995042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6821","nonce":356995042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6825","nonce":356995042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6823","nonce":356995042}]},"public_addr":"172.21.15.186:6819/356995042","cluster_addr":"172.21.15.186:6821/356995042","heartbeat_back_addr":"172.21.15.186:6825/356995042","heartbeat_front_addr":"172.21.15.186:6823/356995042","state":["exists","up"]},{"osd":5,"uuid":"d07ad403-291f-4bde-a64a-5181fb79e544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6811","nonce":3728813703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6813","nonce":3728813703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6817","nonce":3728813703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6815","nonce":3728813703}]},"public_addr":"172.21.15.186:6811/3728813703","cluster_addr":"172.21.15.186:6813/3728813703","heartbeat_back_addr":"172.21.15.186:6817/3728813703","heartbeat_front_addr":"172.21.15.186:6815/3728813703","state":["exists","up"]},{"osd":6,"uuid":"5110d1ad-5634-4c8e-9338-d6c4229b1d63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6803","nonce":4252462313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6805","nonce":4252462313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6809","nonce":4252462313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6807","nonce":4252462313}]},"public_addr":"172.21.15.186:6803/4252462313","cluster_addr":"172.21.15.186:6805/4252462313","heartbeat_back_addr":"172.21.15.186:6809/4252462313","heartbeat_front_addr":"172.21.15.186:6807/4252462313","state":["exists","up"]},{"osd":7,"uuid":"b307b905-5cc8-447b-8765-d888ed6f655b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6827","nonce":1403900459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6829","nonce":1403900459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6833","nonce":1403900459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6831","nonce":1403900459}]},"public_addr":"172.21.15.186:6827/1403900459","cluster_addr":"172.21.15.186:6829/1403900459","heartbeat_back_addr":"172.21.15.186:6833/1403900459","heartbeat_front_addr":"172.21.15.186:6831/1403900459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:01.075527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.428399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:44:59.984305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.439608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:45:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 10 librados: watch_flush enter 2024-05-04T17:45:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 10 librados: watch_flush exit 2024-05-04T17:45:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 20 client.4288.objecter shutdown clearing up homeless session... 2024-05-04T17:45:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 10 client.4288.objecter successfully canceled tick 2024-05-04T17:45:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 1 -- 172.21.15.19:0/2764529440 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f8ea406d810 msgr2=0x7f8ea406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 1 --2- 172.21.15.19:0/2764529440 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f8ea406d810 0x7f8ea406fcd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc006fd0 tx=0x7f8ebc008040 comp rx=0 tx=0).stop 2024-05-04T17:45:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 10 monclient: shutdown 2024-05-04T17:45:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 1 -- 172.21.15.19:0/2764529440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec00a9ac0 msgr2=0x7f8ec0144190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 1 --2- 172.21.15.19:0/2764529440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec00a9ac0 0x7f8ec0144190 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000c430 tx=0x7f8eb000c8f0 comp rx=0 tx=0).stop 2024-05-04T17:45:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 1 -- 172.21.15.19:0/2764529440 shutdown_connections 2024-05-04T17:45:13.022 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 1 --2- 172.21.15.19:0/2764529440 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f8ea406d810 0x7f8ea406fcd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:13.023 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 1 --2- 172.21.15.19:0/2764529440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8ec0143610 0x7f8ec01446d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:13.023 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 1 --2- 172.21.15.19:0/2764529440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ec00a9ac0 0x7f8ec0144190 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:13.023 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 1 --2- 172.21.15.19:0/2764529440 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8ec00a96e0 0x7f8ec01db4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:13.023 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 1 -- 172.21.15.19:0/2764529440 >> 172.21.15.19:0/2764529440 conn(0x7f8ec00678d0 msgr2=0x7f8ec013f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:13.023 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 1 -- 172.21.15.19:0/2764529440 shutdown_connections 2024-05-04T17:45:13.023 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.020+0000 7f8ec72a7640 1 -- 172.21.15.19:0/2764529440 wait complete. 2024-05-04T17:45:13.023 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:13.021+0000 7f8ec72a7640 1 librados: shutdown 2024-05-04T17:45:13.036 DEBUG:teuthology.misc:6 of 8 OSDs are up 2024-05-04T17:45:13.128 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:45:12.959+0000 5dce500 -1 bdev(0x18e7e300 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-04T17:45:13.139 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:45:13.079+0000 5dce500 -1 bdev(0x18e7e300 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-04T17:45:13.338 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:13.333+0000 ac84640 -1 osd.0 0 waiting for initial osdmap 2024-05-04T17:45:13.404 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:45:13.115+0000 5dce500 -1 bdev(0x18e7e300 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-04T17:45:13.407 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:45:13.127+0000 5dce500 -1 bdev(0x18e7e300 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-04T17:45:13.407 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:45:13.140+0000 5dce500 -1 bdev(0x18e7e300 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-04T17:45:13.555 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:45:13.155+0000 5dce500 -1 bdev(0x18e7e300 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-04T17:45:13.572 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:45:13.407+0000 5dce500 -1 bdev(0x18e7e300 /var/lib/ceph/osd/ceph-1/block) aio_submit retries 1 2024-05-04T17:45:13.872 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:45:13.857+0000 14268640 -1 osd.0 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:45:14.634 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:09.565 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:14.721 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:09.610 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:14.772 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:45:14.767+0000 c49a640 -1 osd.1 0 waiting for initial osdmap 2024-05-04T17:45:14.779 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:09.668 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:14.783 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:09.714 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:14.792 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:09.723 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:14.895 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:09.826 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:15.015 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:09.904 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:15.070 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:10.001 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:15.164 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:10.095 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:15.302 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:45:15.287+0000 14268640 -1 osd.1 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-04T17:45:19.844 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-04T17:45:19.937 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.936+0000 7fa675f4b640 10 monclient: get_monmap_and_config 2024-05-04T17:45:19.937 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.936+0000 7fa675f4b640 10 monclient: build_initial_monmap 2024-05-04T17:45:19.937 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.936+0000 7fa675f4b640 10 monclient: monmap: 2024-05-04T17:45:19.937 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:45:19.937 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:19.937 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:45:19.937507+0000 2024-05-04T17:45:19.938 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:45:19.937507+0000 2024-05-04T17:45:19.938 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:19.938 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:19.938 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:19.938 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:19.938 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:19.938 INFO:teuthology.misc.health.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:19.938 INFO:teuthology.misc.health.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:19.938 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:19.940 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.937+0000 7fa675f4b640 1 Processor -- start 2024-05-04T17:45:19.940 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.937+0000 7fa675f4b640 1 -- start start 2024-05-04T17:45:19.940 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.937+0000 7fa675f4b640 10 monclient: init 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fa6700a96e0 0x7fa6701430d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 10 monclient(hunting): picked mon.noname-a con 0x7fa6700a96e0 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6700a9ac0 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa670143610 0x7fa670146840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 10 monclient(hunting): picked mon.noname-b con 0x7fa670143610 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6700ab910 con 0x7fa6700a9ac0 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fa6700abbf0 con 0x7fa6700a96e0 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fa670142bb0 con 0x7fa670143610 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa675f4b640 10 monclient(hunting): authenticate will time out at 1342.051725s 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa66f7fe640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fa6700a96e0 0x7fa6701430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:19.941 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa66ffff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa670143610 0x7fa670146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa66ffff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa670143610 0x7fa670146840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:44498/0 (socket says 172.21.15.19:44498) 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.938+0000 7fa66ffff640 1 -- 172.21.15.19:0/1970557726 learned_addr learned my addr 172.21.15.19:0/1970557726 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66ffff640 10 monclient(hunting): get_auth_request con 0x7fa670143610 auth_method 0 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66ffff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66f7fe640 10 monclient(hunting): get_auth_request con 0x7fa6700a96e0 auth_method 0 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:19.942 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:19.943 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:19.943 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.939+0000 7fa66f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:19.943 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.941+0000 7fa66ffff640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-05-04T17:45:19.943 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.941+0000 7fa66ffff640 1 -- 172.21.15.19:0/1970557726 >> v2:172.21.15.19:3300/0 conn(0x7fa6700a96e0 msgr2=0x7fa6701430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:19.943 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.941+0000 7fa66ffff640 1 --2- 172.21.15.19:0/1970557726 >> v2:172.21.15.19:3300/0 conn(0x7fa6700a96e0 0x7fa6701430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:19.943 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.941+0000 7fa66ffff640 1 -- 172.21.15.19:0/1970557726 >> v1:172.21.15.19:6789/0 conn(0x7fa6700a9ac0 legacy=0x7fa670146130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:19.943 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.941+0000 7fa66ffff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:19.943 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.941+0000 7fa66ffff640 1 monclient: found mon.noname-b 2024-05-04T17:45:19.944 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.941+0000 7fa66ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:19.944 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.941+0000 7fa66ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:45:19.944 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.941+0000 7fa66ffff640 1 -- 172.21.15.19:0/1970557726 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa670143aa0 con 0x7fa670143610 2024-05-04T17:45:19.944 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.943+0000 7fa66ffff640 1 --2- 172.21.15.19:0/1970557726 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa670143610 0x7fa670146840 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa668009e50 tx=0x7fa66802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61f666e067e0097a server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:19.949 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 1 -- 172.21.15.19:0/1970557726 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66802cb00 con 0x7fa670143610 2024-05-04T17:45:19.949 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:19.949 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:19.949 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 10 monclient: dump: 2024-05-04T17:45:19.949 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:45:19.949 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:19.949 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:19.949 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:19.949 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:19.949 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 10 monclient: _finish_auth 0 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 10 monclient: _check_auth_tickets 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 1 -- 172.21.15.19:0/1970557726 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa66802cca0 con 0x7fa670143610 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 1 -- 172.21.15.19:0/1970557726 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa668034610 con 0x7fa670143610 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa675f4b640 5 monclient: authenticate success, global_id 4229 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa675f4b640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa675f4b640 4 set_mon_vals no callback set 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:19.950 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.947+0000 7fa66e7fc640 10 monclient: dump: 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.948+0000 7fa675f4b640 10 monclient: shutdown 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.948+0000 7fa675f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.948+0000 7fa675f4b640 1 -- 172.21.15.19:0/1970557726 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa670143610 msgr2=0x7fa670146840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:19.951 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.948+0000 7fa675f4b640 1 --2- 172.21.15.19:0/1970557726 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa670143610 0x7fa670146840 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa668009e50 tx=0x7fa66802bf60 comp rx=0 tx=0).stop 2024-05-04T17:45:19.952 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.948+0000 7fa675f4b640 1 -- 172.21.15.19:0/1970557726 shutdown_connections 2024-05-04T17:45:19.952 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.948+0000 7fa675f4b640 1 --2- 172.21.15.19:0/1970557726 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa670143610 0x7fa670146840 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:19.952 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.948+0000 7fa675f4b640 1 --2- 172.21.15.19:0/1970557726 >> v2:172.21.15.19:3300/0 conn(0x7fa6700a96e0 0x7fa6701430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:19.952 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.948+0000 7fa675f4b640 1 -- 172.21.15.19:0/1970557726 >> 172.21.15.19:0/1970557726 conn(0x7fa6700678d0 msgr2=0x7fa6700b4fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:19.952 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.948+0000 7fa675f4b640 1 -- 172.21.15.19:0/1970557726 shutdown_connections 2024-05-04T17:45:19.952 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.948+0000 7fa675f4b640 1 -- 172.21.15.19:0/1970557726 wait complete. 2024-05-04T17:45:19.955 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.949+0000 7fa675f4b640 10 monclient: build_initial_monmap 2024-05-04T17:45:19.955 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.949+0000 7fa675f4b640 10 monclient: monmap: 2024-05-04T17:45:19.955 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-05-04T17:45:19.955 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:19.955 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:19.955 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-05-04T17:45:19.955 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:19.955 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:19.955 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:19.955 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:19.955 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.949+0000 7fa675f4b640 1 librados: starting msgr at 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.949+0000 7fa675f4b640 1 librados: starting objecter 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.949+0000 7fa675f4b640 1 Processor -- start 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.949+0000 7fa675f4b640 1 -- start start 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.949+0000 7fa675f4b640 1 librados: setting wanted keys 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.949+0000 7fa675f4b640 1 librados: calling monclient init 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.949+0000 7fa675f4b640 10 monclient: init 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6700a96e0 0x7fa670142540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 10 monclient(hunting): picked mon.noname-b con 0x7fa6700a96e0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa6700a9ac0 0x7fa6701db6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 10 monclient(hunting): picked mon.noname-c con 0x7fa6700a9ac0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa670143610 0x7fa6701dbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 10 monclient(hunting): picked mon.noname-a con 0x7fa670143610 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:19.956 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa6700aaaa0 con 0x7fa670143610 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6700aa920 con 0x7fa6700a96e0 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fa670146d80 con 0x7fa6700a9ac0 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa675f4b640 10 monclient(hunting): authenticate will time out at 1342.064169s 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.950+0000 7fa66effd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa6700a9ac0 0x7fa6701db6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66ffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa670143610 0x7fa6701dbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6700a96e0 0x7fa670142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66effd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa6700a9ac0 0x7fa6701db6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:44512/0 (socket says 172.21.15.19:44512) 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66effd640 1 -- 172.21.15.19:0/1985370262 learned_addr learned my addr 172.21.15.19:0/1985370262 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66effd640 10 monclient(hunting): get_auth_request con 0x7fa6700a9ac0 auth_method 0 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66effd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:19.957 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66f7fe640 10 monclient(hunting): get_auth_request con 0x7fa6700a96e0 auth_method 0 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66ffff640 10 monclient(hunting): get_auth_request con 0x7fa670143610 auth_method 0 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66ffff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:19.958 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.951+0000 7fa66f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.952+0000 7fa66ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.952+0000 7fa66ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.952+0000 7fa66ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.954+0000 7fa66effd640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.954+0000 7fa66effd640 1 -- 172.21.15.19:0/1985370262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6700a96e0 msgr2=0x7fa670142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.954+0000 7fa66effd640 1 --2- 172.21.15.19:0/1985370262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6700a96e0 0x7fa670142540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.954+0000 7fa66effd640 1 -- 172.21.15.19:0/1985370262 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa670143610 msgr2=0x7fa6701dbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.954+0000 7fa66effd640 1 --2- 172.21.15.19:0/1985370262 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa670143610 0x7fa6701dbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.954+0000 7fa66effd640 10 monclient: _finish_hunting 0 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.954+0000 7fa66effd640 1 monclient: found mon.noname-c 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.954+0000 7fa66effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.954+0000 7fa66effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T17:45:19.959 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.954+0000 7fa66effd640 1 -- 172.21.15.19:0/1985370262 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa670144cf0 con 0x7fa6700a9ac0 2024-05-04T17:45:19.960 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.956+0000 7fa66effd640 1 --2- 172.21.15.19:0/1985370262 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa6700a9ac0 0x7fa6701db6e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa66400c430 tx=0x7fa66400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:19.960 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.956+0000 7fa657fff640 10 client.?.objecter ms_handle_connect 0x7fa6700a9ac0 2024-05-04T17:45:19.960 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.956+0000 7fa657fff640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:19.961 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 1 -- 172.21.15.19:0/1985370262 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6640052d0 con 0x7fa6700a9ac0 2024-05-04T17:45:19.961 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:19.961 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:19.961 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 10 monclient: dump: 2024-05-04T17:45:19.961 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:45:19.961 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:19.961 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:19.961 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 10 monclient: _finish_auth 0 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 10 monclient: _check_auth_tickets 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 1 -- 172.21.15.19:0/1985370262 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa664005490 con 0x7fa6700a9ac0 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 1 -- 172.21.15.19:0/1985370262 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664007260 con 0x7fa6700a9ac0 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:19.962 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa657fff640 10 monclient: dump: 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr: 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa675f4b640 5 monclient: authenticate success, global_id 4232 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa675f4b640 10 monclient: _renew_subs 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa675f4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa675f4b640 1 -- 172.21.15.19:0/1985370262 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa670144fd0 con 0x7fa6700a9ac0 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa66dffb640 4 set_mon_vals no callback set 2024-05-04T17:45:19.963 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa675f4b640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:19.964 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa675f4b640 10 monclient: _renew_subs 2024-05-04T17:45:19.964 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa675f4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:19.964 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa675f4b640 1 -- 172.21.15.19:0/1985370262 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa670144310 con 0x7fa6700a9ac0 2024-05-04T17:45:19.964 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.960+0000 7fa675f4b640 1 librados: init done 2024-05-04T17:45:19.964 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.961+0000 7fa675f4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:19.964 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.961+0000 7fa675f4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:19.964 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.961+0000 7fa675f4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:19.965 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:19.961+0000 7fa675f4b640 1 -- 172.21.15.19:0/1985370262 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa67004f5f0 con 0x7fa6700a9ac0 2024-05-04T17:45:20.237 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.236+0000 7fa657fff640 1 -- 172.21.15.19:0/1985370262 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa664007400 con 0x7fa6700a9ac0 2024-05-04T17:45:20.237 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.237+0000 7fa657fff640 1 --2- 172.21.15.19:0/1985370262 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa65c06d810 0x7fa65c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:20.238 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.237+0000 7fa66f7fe640 1 --2- 172.21.15.19:0/1985370262 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa65c06d810 0x7fa65c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:20.238 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.237+0000 7fa66f7fe640 10 monclient: get_auth_request con 0x7fa65c06d810 auth_method 0 2024-05-04T17:45:20.238 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.238+0000 7fa657fff640 10 client.4232.objecter ms_handle_connect 0x7fa65c06d810 2024-05-04T17:45:20.238 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.238+0000 7fa66f7fe640 1 --2- 172.21.15.19:0/1985370262 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa65c06d810 0x7fa65c06fcd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa660009b40 tx=0x7fa660009190 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.252+0000 7fa657fff640 1 -- 172.21.15.19:0/1985370262 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 4857+0+0 (secure 0 0 0) 0x7fa66405f340 con 0x7fa6700a9ac0 2024-05-04T17:45:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.252+0000 7fa657fff640 10 client.4232.objecter ms_dispatch 0x7fa670000b90 osd_map(22..22 src has 1..22) 2024-05-04T17:45:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.252+0000 7fa657fff640 3 client.4232.objecter handle_osd_map got epochs [22,22] > 0 2024-05-04T17:45:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.252+0000 7fa657fff640 3 client.4232.objecter handle_osd_map decoding full epoch 22 2024-05-04T17:45:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.252+0000 7fa657fff640 20 client.4232.objecter dump_active .. 0 homeless 2024-05-04T17:45:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.252+0000 7fa657fff640 1 -- 172.21.15.19:0/1985370262 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa664058b50 con 0x7fa6700a9ac0 2024-05-04T17:45:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.252+0000 7fa657fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.252+0000 7fa657fff640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:20.439 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.438+0000 7fa675f4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:20.439 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.438+0000 7fa675f4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:20.439 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.438+0000 7fa675f4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:20.439 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.438+0000 7fa675f4b640 1 -- 172.21.15.19:0/1985370262 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa67013e530 con 0x7fa6700a9ac0 2024-05-04T17:45:20.563 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.562+0000 7fa657fff640 1 -- 172.21.15.19:0/1985370262 <== mon.2 v2:172.21.15.19:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+12885 (secure 0 0 0) 0x7fa66405ce50 con 0x7fa6700a9ac0 2024-05-04T17:45:20.563 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.562+0000 7fa657fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:20.563 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.562+0000 7fa657fff640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:20.564 INFO:teuthology.misc.health.smithi019.stdout: 2024-05-04T17:45:20.564 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":22,"fsid":"18ced2df-a29f-46cc-b71a-1855bfb60c47","created":"2024-05-04T17:44:16.995866+0000","modified":"2024-05-04T17:45:17.136154+0000","last_up_change":"2024-05-04T17:45:15.621507+0000","last_in_change":"2024-05-04T17:44:23.464900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T17:45:03.247977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b749ec0-22b6-458c-a7af-2ccace806ab7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6809","nonce":2236557705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6811","nonce":2236557705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6815","nonce":2236557705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6813","nonce":2236557705}]},"public_addr":"172.21.15.19:6809/2236557705","cluster_addr":"172.21.15.19:6811/2236557705","heartbeat_back_addr":"172.21.15.19:6815/2236557705","heartbeat_front_addr":"172.21.15.19:6813/2236557705","state":["exists","up"]},{"osd":1,"uuid":"3aa520fe-39c4-4fa2-860b-f4d0cc04ca97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6801","nonce":3360656308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6803","nonce":3360656308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6807","nonce":3360656308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6805","nonce":3360656308}]},"public_addr":"172.21.15.19:6801/3360656308","cluster_addr":"172.21.15.19:6803/3360656308","heartbeat_back_addr":"172.21.15.19:6807/3360656308","heartbeat_front_addr":"172.21.15.19:6805/3360656308","state":["exists","up"]},{"osd":2,"uuid":"16ed21c9-e5c3-4d65-b4d1-e4927a0fe919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6817","nonce":3744614279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6820","nonce":3744614279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6825","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6827","nonce":3744614279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6823","nonce":3744614279}]},"public_addr":"172.21.15.19:6817/3744614279","cluster_addr":"172.21.15.19:6820/3744614279","heartbeat_back_addr":"172.21.15.19:6827/3744614279","heartbeat_front_addr":"172.21.15.19:6823/3744614279","state":["exists","up"]},{"osd":3,"uuid":"45aa2c0f-cfae-478c-a60c-4c1491e45525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6819","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6821","nonce":2537429684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6826","nonce":2537429684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6831","nonce":2537429684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6829","nonce":2537429684}]},"public_addr":"172.21.15.19:6821/2537429684","cluster_addr":"172.21.15.19:6826/2537429684","heartbeat_back_addr":"172.21.15.19:6831/2537429684","heartbeat_front_addr":"172.21.15.19:6829/2537429684","state":["exists","up"]},{"osd":4,"uuid":"cb00f84f-d78c-461c-9843-73a9da4b80a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6819","nonce":356995042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6821","nonce":356995042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6825","nonce":356995042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6823","nonce":356995042}]},"public_addr":"172.21.15.186:6819/356995042","cluster_addr":"172.21.15.186:6821/356995042","heartbeat_back_addr":"172.21.15.186:6825/356995042","heartbeat_front_addr":"172.21.15.186:6823/356995042","state":["exists","up"]},{"osd":5,"uuid":"d07ad403-291f-4bde-a64a-5181fb79e544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6811","nonce":3728813703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6813","nonce":3728813703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6817","nonce":3728813703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6815","nonce":3728813703}]},"public_addr":"172.21.15.186:6811/3728813703","cluster_addr":"172.21.15.186:6813/3728813703","heartbeat_back_addr":"172.21.15.186:6817/3728813703","heartbeat_front_addr":"172.21.15.186:6815/3728813703","state":["exists","up"]},{"osd":6,"uuid":"5110d1ad-5634-4c8e-9338-d6c4229b1d63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6803","nonce":4252462313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6805","nonce":4252462313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6809","nonce":4252462313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6807","nonce":4252462313}]},"public_addr":"172.21.15.186:6803/4252462313","cluster_addr":"172.21.15.186:6805/4252462313","heartbeat_back_addr":"172.21.15.186:6809/4252462313","heartbeat_front_addr":"172.21.15.186:6807/4252462313","state":["exists","up"]},{"osd":7,"uuid":"b307b905-5cc8-447b-8765-d888ed6f655b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6827","nonce":1403900459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6829","nonce":1403900459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6833","nonce":1403900459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6831","nonce":1403900459}]},"public_addr":"172.21.15.186:6827/1403900459","cluster_addr":"172.21.15.186:6829/1403900459","heartbeat_back_addr":"172.21.15.186:6833/1403900459","heartbeat_front_addr":"172.21.15.186:6831/1403900459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.360122+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.570945+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:01.075527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.428399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.550872+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:44:59.984305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.443005+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.439608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:45:20.566 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.564+0000 7fa675f4b640 10 librados: watch_flush enter 2024-05-04T17:45:20.566 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.564+0000 7fa675f4b640 10 librados: watch_flush exit 2024-05-04T17:45:20.566 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.564+0000 7fa675f4b640 20 client.4232.objecter shutdown clearing up homeless session... 2024-05-04T17:45:20.566 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.564+0000 7fa675f4b640 10 client.4232.objecter successfully canceled tick 2024-05-04T17:45:20.566 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 -- 172.21.15.19:0/1985370262 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa65c06d810 msgr2=0x7fa65c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:20.566 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 --2- 172.21.15.19:0/1985370262 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa65c06d810 0x7fa65c06fcd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa660009b40 tx=0x7fa660009190 comp rx=0 tx=0).stop 2024-05-04T17:45:20.566 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 10 monclient: shutdown 2024-05-04T17:45:20.566 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:20.567 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 -- 172.21.15.19:0/1985370262 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa6700a9ac0 msgr2=0x7fa6701db6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:20.567 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 --2- 172.21.15.19:0/1985370262 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa6700a9ac0 0x7fa6701db6e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa66400c430 tx=0x7fa66400c8f0 comp rx=0 tx=0).stop 2024-05-04T17:45:20.567 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 -- 172.21.15.19:0/1985370262 shutdown_connections 2024-05-04T17:45:20.567 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 --2- 172.21.15.19:0/1985370262 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa65c06d810 0x7fa65c06fcd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:20.567 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 --2- 172.21.15.19:0/1985370262 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa670143610 0x7fa6701dbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:20.567 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 --2- 172.21.15.19:0/1985370262 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa6700a9ac0 0x7fa6701db6e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:20.567 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 --2- 172.21.15.19:0/1985370262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6700a96e0 0x7fa670142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:20.567 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 -- 172.21.15.19:0/1985370262 >> 172.21.15.19:0/1985370262 conn(0x7fa6700678d0 msgr2=0x7fa670140180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:20.567 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 -- 172.21.15.19:0/1985370262 shutdown_connections 2024-05-04T17:45:20.567 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 -- 172.21.15.19:0/1985370262 wait complete. 2024-05-04T17:45:20.567 INFO:teuthology.misc.health.smithi019.stderr:2024-05-04T17:45:20.565+0000 7fa675f4b640 1 librados: shutdown 2024-05-04T17:45:20.576 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-04T17:45:20.576 INFO:tasks.ceph:Creating RBD pool 2024-05-04T17:45:20.576 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd pool create rbd 8 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.690+0000 7fa67473a640 10 monclient: get_monmap_and_config 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.690+0000 7fa67473a640 10 monclient: build_initial_monmap 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.690+0000 7fa67473a640 10 monclient: monmap: 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:20.691140+0000 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:20.691140+0000 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:20.691 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:20.692 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:20.692 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:20.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.690+0000 7fa67473a640 1 Processor -- start 2024-05-04T17:45:20.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 1 -- start start 2024-05-04T17:45:20.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 10 monclient: init 2024-05-04T17:45:20.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:20.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-05-04T17:45:20.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fa66c0a9720 0x7fa66c0aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:20.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 10 monclient(hunting): picked mon.noname-a con 0x7fa66c0a9720 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:20.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa66c0a9b00 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:20.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa66c0b2e90 0x7fa66c13fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:20.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 10 monclient(hunting): picked mon.noname-c con 0x7fa66c0b2e90 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:20.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa66c0abc40 con 0x7fa66c0a9b00 2024-05-04T17:45:20.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:20.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fa66c146fa0 con 0x7fa66c0a9720 2024-05-04T17:45:20.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:20.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa66c146090 con 0x7fa66c0b2e90 2024-05-04T17:45:20.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:20.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.691+0000 7fa67473a640 10 monclient(hunting): authenticate will time out at 1342.805512s 2024-05-04T17:45:20.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.692+0000 7fa672cd7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa66c0b2e90 0x7fa66c13fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:20.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.693+0000 7fa671cd5640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fa66c0a9b00 0x7fa66c146890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:46400/0 (socket says 172.21.15.19:46400) 2024-05-04T17:45:20.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.693+0000 7fa671cd5640 1 -- 172.21.15.19:0/3250991197 learned_addr learned my addr 172.21.15.19:0/3250991197 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:20.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.693+0000 7fa672cd7640 10 monclient(hunting): get_auth_request con 0x7fa66c0b2e90 auth_method 0 2024-05-04T17:45:20.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.693+0000 7fa672cd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:20.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.693+0000 7fa672cd7640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:20.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.693+0000 7fa672cd7640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:20.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.694+0000 7fa672cd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:20.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.694+0000 7fa672cd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:20.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.694+0000 7fa672cd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:20.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.696+0000 7fa672cd7640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-05-04T17:45:20.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.696+0000 7fa672cd7640 1 -- 172.21.15.19:0/3250991197 >> v2:172.21.15.19:3300/0 conn(0x7fa66c0a9720 msgr2=0x7fa66c0aa890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:20.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.696+0000 7fa672cd7640 1 --2- 172.21.15.19:0/3250991197 >> v2:172.21.15.19:3300/0 conn(0x7fa66c0a9720 0x7fa66c0aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:20.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.696+0000 7fa672cd7640 1 -- 172.21.15.19:0/3250991197 >> v1:172.21.15.186:6789/0 conn(0x7fa66c0a9b00 legacy=0x7fa66c146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:20.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.696+0000 7fa672cd7640 10 monclient: _finish_hunting 0 2024-05-04T17:45:20.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.696+0000 7fa672cd7640 1 monclient: found mon.noname-c 2024-05-04T17:45:20.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.696+0000 7fa672cd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:20.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.696+0000 7fa672cd7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:20.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.696+0000 7fa672cd7640 1 -- 172.21.15.19:0/3250991197 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66c146370 con 0x7fa66c0b2e90 2024-05-04T17:45:20.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.700+0000 7fa672cd7640 1 --2- 172.21.15.19:0/3250991197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa66c0b2e90 0x7fa66c13fea0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa664009d90 tx=0x7fa66402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb5fbd4fa78f7684 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:20.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 1 -- 172.21.15.19:0/3250991197 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664039070 con 0x7fa66c0b2e90 2024-05-04T17:45:20.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:20.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:20.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 10 monclient: dump: 2024-05-04T17:45:20.707 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 10 monclient: _finish_auth 0 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 10 monclient: _check_auth_tickets 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 1 -- 172.21.15.19:0/3250991197 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa66402cc60 con 0x7fa66c0b2e90 2024-05-04T17:45:20.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 1 -- 172.21.15.19:0/3250991197 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664034660 con 0x7fa66c0b2e90 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa6714d4640 10 monclient: dump: 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:20.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa67473a640 5 monclient: authenticate success, global_id 4297 2024-05-04T17:45:20.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa67473a640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:20.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.706+0000 7fa67473a640 4 set_mon_vals no callback set 2024-05-04T17:45:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.707+0000 7fa67473a640 10 monclient: shutdown 2024-05-04T17:45:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.707+0000 7fa67473a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.707+0000 7fa67473a640 1 -- 172.21.15.19:0/3250991197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa66c0b2e90 msgr2=0x7fa66c13fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.707+0000 7fa67473a640 1 --2- 172.21.15.19:0/3250991197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa66c0b2e90 0x7fa66c13fea0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa664009d90 tx=0x7fa66402beb0 comp rx=0 tx=0).stop 2024-05-04T17:45:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.707+0000 7fa67473a640 1 -- 172.21.15.19:0/3250991197 shutdown_connections 2024-05-04T17:45:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.707+0000 7fa67473a640 1 --2- 172.21.15.19:0/3250991197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa66c0b2e90 0x7fa66c13fea0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.707+0000 7fa67473a640 1 --2- 172.21.15.19:0/3250991197 >> v2:172.21.15.19:3300/0 conn(0x7fa66c0a9720 0x7fa66c0aa890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.707+0000 7fa67473a640 1 -- 172.21.15.19:0/3250991197 >> 172.21.15.19:0/3250991197 conn(0x7fa66c067780 msgr2=0x7fa66c0b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.707+0000 7fa67473a640 1 -- 172.21.15.19:0/3250991197 shutdown_connections 2024-05-04T17:45:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.707+0000 7fa67473a640 1 -- 172.21.15.19:0/3250991197 wait complete. 2024-05-04T17:45:20.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.708+0000 7fa67473a640 10 monclient: build_initial_monmap 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.708+0000 7fa67473a640 10 monclient: monmap: 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.708+0000 7fa67473a640 1 librados: starting msgr at 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.708+0000 7fa67473a640 1 librados: starting objecter 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.708+0000 7fa67473a640 1 Processor -- start 2024-05-04T17:45:20.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.708+0000 7fa67473a640 1 -- start start 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.708+0000 7fa67473a640 1 librados: setting wanted keys 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.708+0000 7fa67473a640 1 librados: calling monclient init 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.708+0000 7fa67473a640 10 monclient: init 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa66c0a9720 0x7fa66c1353a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 10 monclient(hunting): picked mon.noname-a con 0x7fa66c0a9720 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa66c0a9b00 0x7fa66c1db5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 10 monclient(hunting): picked mon.noname-c con 0x7fa66c0a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa66c0b2e90 0x7fa66c136a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 10 monclient(hunting): picked mon.noname-b con 0x7fa66c0b2e90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c140770 con 0x7fa66c0a9720 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c1405f0 con 0x7fa66c0b2e90 2024-05-04T17:45:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fa66c14d420 con 0x7fa66c0a9b00 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa67473a640 10 monclient(hunting): authenticate will time out at 1342.822906s 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa671cd5640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa66c0a9b00 0x7fa66c1db5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa6724d6640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa66c0a9720 0x7fa66c1353a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.709+0000 7fa671cd5640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa66c0a9b00 0x7fa66c1db5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:36842/0 (socket says 172.21.15.19:36842) 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa671cd5640 1 -- 172.21.15.19:0/1659396236 learned_addr learned my addr 172.21.15.19:0/1659396236 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa671cd5640 10 monclient(hunting): get_auth_request con 0x7fa66c0a9b00 auth_method 0 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa671cd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa671cd5640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa671cd5640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa672cd7640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa66c0b2e90 0x7fa66c136a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa6724d6640 10 monclient(hunting): get_auth_request con 0x7fa66c0a9720 auth_method 0 2024-05-04T17:45:20.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa6724d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa6724d6640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa6724d6640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa672cd7640 10 monclient(hunting): get_auth_request con 0x7fa66c0b2e90 auth_method 0 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa672cd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa672cd7640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa672cd7640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa671cd5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa671cd5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa671cd5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa6724d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa6724d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa6724d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa672cd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa672cd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:20.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.710+0000 7fa672cd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:20.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.713+0000 7fa672cd7640 10 monclient(hunting): handle_auth_done global_id 4300 payload 824 2024-05-04T17:45:20.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.713+0000 7fa672cd7640 1 -- 172.21.15.19:0/1659396236 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa66c0a9b00 msgr2=0x7fa66c1db5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:20.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.713+0000 7fa672cd7640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa66c0a9b00 0x7fa66c1db5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.713+0000 7fa672cd7640 1 -- 172.21.15.19:0/1659396236 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa66c0a9720 msgr2=0x7fa66c1353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.713+0000 7fa672cd7640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa66c0a9720 0x7fa66c1353a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.713+0000 7fa672cd7640 10 monclient: _finish_hunting 0 2024-05-04T17:45:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.713+0000 7fa672cd7640 1 monclient: found mon.noname-b 2024-05-04T17:45:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.713+0000 7fa672cd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.713+0000 7fa672cd7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.713+0000 7fa672cd7640 1 -- 172.21.15.19:0/1659396236 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66c137fe0 con 0x7fa66c0b2e90 2024-05-04T17:45:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.716+0000 7fa672cd7640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa66c0b2e90 0x7fa66c136a10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0b91d0 tx=0x7fa664004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.716+0000 7fa662ffd640 10 client.?.objecter ms_handle_connect 0x7fa66c0b2e90 2024-05-04T17:45:20.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.716+0000 7fa662ffd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:20.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 1 -- 172.21.15.19:0/1659396236 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664039050 con 0x7fa66c0b2e90 2024-05-04T17:45:20.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:20.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 10 monclient: dump: 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 10 monclient: _finish_auth 0 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 10 monclient: _check_auth_tickets 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:20.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 1 -- 172.21.15.19:0/1659396236 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa664045070 con 0x7fa66c0b2e90 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 1 -- 172.21.15.19:0/1659396236 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66403e850 con 0x7fa66c0b2e90 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa662ffd640 10 monclient: dump: 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:20.722 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:20.723 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:20.723 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:20.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa67473a640 5 monclient: authenticate success, global_id 4300 2024-05-04T17:45:20.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa67473a640 10 monclient: _renew_subs 2024-05-04T17:45:20.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa67473a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:20.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa670cd3640 4 set_mon_vals no callback set 2024-05-04T17:45:20.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa67473a640 1 -- 172.21.15.19:0/1659396236 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66c0b7650 con 0x7fa66c0b2e90 2024-05-04T17:45:20.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa67473a640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:20.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa67473a640 10 monclient: _renew_subs 2024-05-04T17:45:20.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa67473a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:20.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa67473a640 1 -- 172.21.15.19:0/1659396236 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa66c137130 con 0x7fa66c0b2e90 2024-05-04T17:45:20.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.719+0000 7fa67473a640 1 librados: init done 2024-05-04T17:45:20.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.720+0000 7fa67473a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:20.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.720+0000 7fa67473a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:20.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.720+0000 7fa67473a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:20.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.720+0000 7fa67473a640 1 -- 172.21.15.19:0/1659396236 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa66c06a700 con 0x7fa66c0b2e90 2024-05-04T17:45:20.739 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.739+0000 7fa662ffd640 1 -- 172.21.15.19:0/1659396236 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa66403e9f0 con 0x7fa66c0b2e90 2024-05-04T17:45:20.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.739+0000 7fa662ffd640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa65006d810 0x7fa65006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:20.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.739+0000 7fa6724d6640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa65006d810 0x7fa65006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:20.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.739+0000 7fa6724d6640 10 monclient: get_auth_request con 0x7fa65006d810 auth_method 0 2024-05-04T17:45:20.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.740+0000 7fa6724d6640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa65006d810 0x7fa65006fcd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa668009990 tx=0x7fa668009110 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:20.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.740+0000 7fa662ffd640 10 client.4300.objecter ms_handle_connect 0x7fa65006d810 2024-05-04T17:45:20.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.741+0000 7fa662ffd640 1 -- 172.21.15.19:0/1659396236 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 4857+0+0 (secure 0 0 0) 0x7fa664083040 con 0x7fa66c0b2e90 2024-05-04T17:45:20.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.741+0000 7fa662ffd640 10 client.4300.objecter ms_dispatch 0x7fa66c000b90 osd_map(22..22 src has 1..22) 2024-05-04T17:45:20.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.741+0000 7fa662ffd640 3 client.4300.objecter handle_osd_map got epochs [22,22] > 0 2024-05-04T17:45:20.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.741+0000 7fa662ffd640 3 client.4300.objecter handle_osd_map decoding full epoch 22 2024-05-04T17:45:20.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.741+0000 7fa662ffd640 20 client.4300.objecter dump_active .. 0 homeless 2024-05-04T17:45:20.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.982+0000 7fa662ffd640 1 -- 172.21.15.19:0/1659396236 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa66407cea0 con 0x7fa66c0b2e90 2024-05-04T17:45:20.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.982+0000 7fa662ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:20.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:20.982+0000 7fa662ffd640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:21.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.168+0000 7fa67473a640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-05-04T17:45:21.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.168+0000 7fa67473a640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-05-04T17:45:21.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.168+0000 7fa67473a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:21.169 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.168+0000 7fa67473a640 1 -- 172.21.15.19:0/1659396236 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fa66c0b7880 con 0x7fa66c0b2e90 2024-05-04T17:45:21.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.857+0000 7fa662ffd640 1 -- 172.21.15.19:0/1659396236 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v23) ==== 109+0+0 (secure 0 0 0) 0x7fa66407c7a0 con 0x7fa66c0b2e90 2024-05-04T17:45:21.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.857+0000 7fa662ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-05-04T17:45:21.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.857+0000 7fa662ffd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-05-04T17:45:21.858 INFO:teuthology.orchestra.run.smithi019.stderr:pool 'rbd' created 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 10 librados: watch_flush enter 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 10 librados: watch_flush exit 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 20 client.4300.objecter shutdown clearing up homeless session... 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 10 client.4300.objecter successfully canceled tick 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 -- 172.21.15.19:0/1659396236 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa65006d810 msgr2=0x7fa65006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa65006d810 0x7fa65006fcd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa668009990 tx=0x7fa668009110 comp rx=0 tx=0).stop 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 10 monclient: shutdown 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 -- 172.21.15.19:0/1659396236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa66c0b2e90 msgr2=0x7fa66c136a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa66c0b2e90 0x7fa66c136a10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0b91d0 tx=0x7fa664004a40 comp rx=0 tx=0).stop 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 -- 172.21.15.19:0/1659396236 shutdown_connections 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa65006d810 0x7fa65006fcd0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa66c0b2e90 0x7fa66c136a10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:21.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa66c0a9b00 0x7fa66c1db5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:21.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 --2- 172.21.15.19:0/1659396236 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa66c0a9720 0x7fa66c1353a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:21.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 -- 172.21.15.19:0/1659396236 >> 172.21.15.19:0/1659396236 conn(0x7fa66c067780 msgr2=0x7fa66c0aa510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:21.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 -- 172.21.15.19:0/1659396236 shutdown_connections 2024-05-04T17:45:21.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 -- 172.21.15.19:0/1659396236 wait complete. 2024-05-04T17:45:21.862 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.859+0000 7fa67473a640 1 librados: shutdown 2024-05-04T17:45:21.872 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-04T17:45:21.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.973+0000 7fe658523640 10 monclient: get_monmap_and_config 2024-05-04T17:45:21.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.973+0000 7fe658523640 10 monclient: build_initial_monmap 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.973+0000 7fe658523640 10 monclient: monmap: 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:21.974791+0000 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:21.974791+0000 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:21.975 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:21.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.974+0000 7fe658523640 1 Processor -- start 2024-05-04T17:45:21.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.974+0000 7fe658523640 1 -- start start 2024-05-04T17:45:21.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.974+0000 7fe658523640 10 monclient: init 2024-05-04T17:45:21.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.975+0000 7fe658523640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:21.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.975+0000 7fe658523640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-05-04T17:45:21.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.975+0000 7fe658523640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6500a9720 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:21.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.975+0000 7fe658523640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fe6500a9b00 0x7fe6500aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:21.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.975+0000 7fe658523640 10 monclient(hunting): picked mon.noname-a con 0x7fe6500a9b00 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:21.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.975+0000 7fe658523640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6500b2e90 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:21.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.975+0000 7fe658523640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6500abc40 con 0x7fe6500a9720 2024-05-04T17:45:21.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.975+0000 7fe658523640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe650146fa0 con 0x7fe6500b2e90 2024-05-04T17:45:21.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.975+0000 7fe658523640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:21.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.975+0000 7fe658523640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fe650146090 con 0x7fe6500a9b00 2024-05-04T17:45:21.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.975+0000 7fe658523640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:21.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.975+0000 7fe658523640 10 monclient(hunting): authenticate will time out at 1344.089527s 2024-05-04T17:45:21.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.976+0000 7fe655abe640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fe6500a9b00 0x7fe6500aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:21.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.976+0000 7fe6562bf640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fe6500a9720 0x7fe650146890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:42266/0 (socket says 172.21.15.19:42266) 2024-05-04T17:45:21.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.976+0000 7fe6562bf640 1 -- 172.21.15.19:0/2184048499 learned_addr learned my addr 172.21.15.19:0/2184048499 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:21.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.977+0000 7fe655abe640 10 monclient(hunting): get_auth_request con 0x7fe6500a9b00 auth_method 0 2024-05-04T17:45:21.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.977+0000 7fe655abe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:21.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.977+0000 7fe655abe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:21.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.977+0000 7fe655abe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:21.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.977+0000 7fe655abe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:21.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.977+0000 7fe655abe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:21.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.977+0000 7fe655abe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:21.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.979+0000 7fe655abe640 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2024-05-04T17:45:21.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.979+0000 7fe655abe640 1 -- 172.21.15.19:0/2184048499 >> v1:172.21.15.186:6789/0 conn(0x7fe6500b2e90 legacy=0x7fe65013fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:21.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.979+0000 7fe655abe640 1 -- 172.21.15.19:0/2184048499 >> v1:172.21.15.19:6789/0 conn(0x7fe6500a9720 legacy=0x7fe650146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:21.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.979+0000 7fe655abe640 10 monclient: _finish_hunting 0 2024-05-04T17:45:21.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.979+0000 7fe655abe640 1 monclient: found mon.noname-a 2024-05-04T17:45:21.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.979+0000 7fe655abe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:21.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.979+0000 7fe655abe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:21.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.979+0000 7fe655abe640 1 -- 172.21.15.19:0/2184048499 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe650146370 con 0x7fe6500a9b00 2024-05-04T17:45:21.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.983+0000 7fe655abe640 1 --2- 172.21.15.19:0/2184048499 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6500a9b00 0x7fe6500aa890 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe6400097d0 tx=0x7fe64002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1829c6305f2500 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:21.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 1 -- 172.21.15.19:0/2184048499 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe640034070 con 0x7fe6500a9b00 2024-05-04T17:45:21.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:21.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:21.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 10 monclient: dump: 2024-05-04T17:45:21.988 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 10 monclient: _finish_auth 0 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 10 monclient: _check_auth_tickets 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe658523640 5 monclient: authenticate success, global_id 4263 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe658523640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:21.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 1 -- 172.21.15.19:0/2184048499 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe640030610 con 0x7fe6500a9b00 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe658523640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe658523640 4 set_mon_vals no callback set 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 1 -- 172.21.15.19:0/2184048499 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6400309e0 con 0x7fe6500a9b00 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.987+0000 7fe6552bd640 10 monclient: dump: 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:21.990 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:21.991 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:21.991 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.988+0000 7fe658523640 10 monclient: shutdown 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.988+0000 7fe658523640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.988+0000 7fe658523640 1 -- 172.21.15.19:0/2184048499 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6500a9b00 msgr2=0x7fe6500aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.988+0000 7fe658523640 1 --2- 172.21.15.19:0/2184048499 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6500a9b00 0x7fe6500aa890 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe6400097d0 tx=0x7fe64002b960 comp rx=0 tx=0).stop 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.988+0000 7fe658523640 1 -- 172.21.15.19:0/2184048499 shutdown_connections 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.988+0000 7fe658523640 1 --2- 172.21.15.19:0/2184048499 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6500a9b00 0x7fe6500aa890 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.988+0000 7fe658523640 1 -- 172.21.15.19:0/2184048499 >> 172.21.15.19:0/2184048499 conn(0x7fe650067780 msgr2=0x7fe6500b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.988+0000 7fe658523640 1 -- 172.21.15.19:0/2184048499 shutdown_connections 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.988+0000 7fe658523640 1 -- 172.21.15.19:0/2184048499 wait complete. 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.989+0000 7fe658523640 10 monclient: build_initial_monmap 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.989+0000 7fe658523640 10 monclient: monmap: 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:21.994 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.989+0000 7fe658523640 1 librados: starting msgr at 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.989+0000 7fe658523640 1 librados: starting objecter 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.989+0000 7fe658523640 1 Processor -- start 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.989+0000 7fe658523640 1 -- start start 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.989+0000 7fe658523640 1 librados: setting wanted keys 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.989+0000 7fe658523640 1 librados: calling monclient init 2024-05-04T17:45:21.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.989+0000 7fe658523640 10 monclient: init 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6500a9720 0x7fe650145760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 10 monclient(hunting): picked mon.noname-b con 0x7fe6500a9720 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6500a9b00 0x7fe65013dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 10 monclient(hunting): picked mon.noname-a con 0x7fe6500a9b00 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe6500b2e90 0x7fe65013e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 10 monclient(hunting): picked mon.noname-c con 0x7fe6500b2e90 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe65014e430 con 0x7fe6500a9b00 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe65014e5b0 con 0x7fe6500a9720 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fe65014e730 con 0x7fe6500b2e90 2024-05-04T17:45:21.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe658523640 10 monclient(hunting): authenticate will time out at 1344.104083s 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.990+0000 7fe6562bf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6500a9720 0x7fe650145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe656ac0640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe6500b2e90 0x7fe65013e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe6562bf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6500a9720 0x7fe650145760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36190/0 (socket says 172.21.15.19:36190) 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe6562bf640 1 -- 172.21.15.19:0/3410468440 learned_addr learned my addr 172.21.15.19:0/3410468440 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe6562bf640 10 monclient(hunting): get_auth_request con 0x7fe6500a9720 auth_method 0 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe6562bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe6562bf640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe6562bf640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe656ac0640 10 monclient(hunting): get_auth_request con 0x7fe6500b2e90 auth_method 0 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe656ac0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe656ac0640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe656ac0640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe655abe640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6500a9b00 0x7fe65013dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:21.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe6562bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:21.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe6562bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:21.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe6562bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:21.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe656ac0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:21.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe656ac0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:21.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.991+0000 7fe656ac0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:21.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.992+0000 7fe655abe640 10 monclient(hunting): get_auth_request con 0x7fe6500a9b00 auth_method 0 2024-05-04T17:45:21.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.992+0000 7fe655abe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:21.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.992+0000 7fe655abe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:21.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.992+0000 7fe655abe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:21.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.992+0000 7fe655abe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:21.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.992+0000 7fe655abe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:21.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.992+0000 7fe655abe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:21.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.994+0000 7fe6562bf640 10 monclient(hunting): handle_auth_done global_id 4306 payload 824 2024-05-04T17:45:21.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.994+0000 7fe6562bf640 1 -- 172.21.15.19:0/3410468440 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe6500b2e90 msgr2=0x7fe65013e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:21.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.994+0000 7fe6562bf640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe6500b2e90 0x7fe65013e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:21.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.994+0000 7fe6562bf640 1 -- 172.21.15.19:0/3410468440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6500a9b00 msgr2=0x7fe65013dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:22.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.994+0000 7fe6562bf640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6500a9b00 0x7fe65013dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:22.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.994+0000 7fe6562bf640 10 monclient: _finish_hunting 0 2024-05-04T17:45:22.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.994+0000 7fe6562bf640 1 monclient: found mon.noname-b 2024-05-04T17:45:22.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.994+0000 7fe6562bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:22.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.994+0000 7fe6562bf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:22.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.994+0000 7fe6562bf640 1 -- 172.21.15.19:0/3410468440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6501415b0 con 0x7fe6500a9720 2024-05-04T17:45:22.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.996+0000 7fe6562bf640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6500a9720 0x7fe650145760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00c400 tx=0x7fe64c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:22.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.996+0000 7fe646ffd640 10 client.?.objecter ms_handle_connect 0x7fe6500a9720 2024-05-04T17:45:22.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:21.996+0000 7fe646ffd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:22.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 1 -- 172.21.15.19:0/3410468440 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c017070 con 0x7fe6500a9720 2024-05-04T17:45:22.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:22.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:22.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 10 monclient: dump: 2024-05-04T17:45:22.002 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:22.002 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 10 monclient: _finish_auth 0 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 10 monclient: _check_auth_tickets 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 1 -- 172.21.15.19:0/3410468440 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe64c005ce0 con 0x7fe6500a9720 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 1 -- 172.21.15.19:0/3410468440 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0072f0 con 0x7fe6500a9720 2024-05-04T17:45:22.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe658523640 5 monclient: authenticate success, global_id 4306 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe658523640 10 monclient: _renew_subs 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe658523640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe658523640 1 -- 172.21.15.19:0/3410468440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe650141890 con 0x7fe6500a9720 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe654abc640 4 set_mon_vals no callback set 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe646ffd640 10 monclient: dump: 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:22.004 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:22.005 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:22.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe658523640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:22.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe658523640 10 monclient: _renew_subs 2024-05-04T17:45:22.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe658523640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:22.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe658523640 1 -- 172.21.15.19:0/3410468440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe650140950 con 0x7fe6500a9720 2024-05-04T17:45:22.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.001+0000 7fe658523640 1 librados: init done 2024-05-04T17:45:22.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.002+0000 7fe658523640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:22.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.002+0000 7fe658523640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:22.005 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.002+0000 7fe658523640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:22.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.002+0000 7fe658523640 1 -- 172.21.15.19:0/3410468440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe65006a700 con 0x7fe6500a9720 2024-05-04T17:45:22.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.037+0000 7fe646ffd640 1 -- 172.21.15.19:0/3410468440 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe64c01a070 con 0x7fe6500a9720 2024-05-04T17:45:22.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.037+0000 7fe646ffd640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe63406d810 0x7fe63406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:22.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.037+0000 7fe655abe640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe63406d810 0x7fe63406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:22.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.037+0000 7fe655abe640 10 monclient: get_auth_request con 0x7fe63406d810 auth_method 0 2024-05-04T17:45:22.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.038+0000 7fe655abe640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe63406d810 0x7fe63406fcd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe640036000 tx=0x7fe64003b040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:22.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.038+0000 7fe646ffd640 10 client.4306.objecter ms_handle_connect 0x7fe63406d810 2024-05-04T17:45:22.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.316+0000 7fe646ffd640 1 -- 172.21.15.19:0/3410468440 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5199+0+0 (secure 0 0 0) 0x7fe64c056a70 con 0x7fe6500a9720 2024-05-04T17:45:22.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.316+0000 7fe646ffd640 10 client.4306.objecter ms_dispatch 0x7fe650000b90 osd_map(23..23 src has 1..23) 2024-05-04T17:45:22.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.316+0000 7fe646ffd640 3 client.4306.objecter handle_osd_map got epochs [23,23] > 0 2024-05-04T17:45:22.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.316+0000 7fe646ffd640 3 client.4306.objecter handle_osd_map decoding full epoch 23 2024-05-04T17:45:22.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.316+0000 7fe646ffd640 20 client.4306.objecter dump_active .. 0 homeless 2024-05-04T17:45:22.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.336+0000 7fe646ffd640 1 -- 172.21.15.19:0/3410468440 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe64c021020 con 0x7fe6500a9720 2024-05-04T17:45:22.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.336+0000 7fe646ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:22.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.336+0000 7fe646ffd640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:22.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.568+0000 7fe658523640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-05-04T17:45:22.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.568+0000 7fe658523640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-05-04T17:45:22.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.568+0000 7fe658523640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:22.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.568+0000 7fe658523640 1 -- 172.21.15.19:0/3410468440 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7fe650073120 con 0x7fe6500a9720 2024-05-04T17:45:22.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.945+0000 7fe646ffd640 1 -- 172.21.15.19:0/3410468440 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v24) ==== 173+0+0 (secure 0 0 0) 0x7fe64c0589d0 con 0x7fe6500a9720 2024-05-04T17:45:22.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.945+0000 7fe646ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-05-04T17:45:22.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.945+0000 7fe646ffd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-05-04T17:45:22.946 INFO:teuthology.orchestra.run.smithi019.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-04T17:45:22.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.947+0000 7fe658523640 10 librados: watch_flush enter 2024-05-04T17:45:22.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.947+0000 7fe658523640 10 librados: watch_flush exit 2024-05-04T17:45:22.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.947+0000 7fe658523640 20 client.4306.objecter shutdown clearing up homeless session... 2024-05-04T17:45:22.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.947+0000 7fe658523640 10 client.4306.objecter successfully canceled tick 2024-05-04T17:45:22.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 -- 172.21.15.19:0/3410468440 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe63406d810 msgr2=0x7fe63406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:22.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe63406d810 0x7fe63406fcd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe640036000 tx=0x7fe64003b040 comp rx=0 tx=0).stop 2024-05-04T17:45:22.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 10 monclient: shutdown 2024-05-04T17:45:22.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:22.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 -- 172.21.15.19:0/3410468440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6500a9720 msgr2=0x7fe650145760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:22.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6500a9720 0x7fe650145760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00c400 tx=0x7fe64c00c8c0 comp rx=0 tx=0).stop 2024-05-04T17:45:22.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 -- 172.21.15.19:0/3410468440 shutdown_connections 2024-05-04T17:45:22.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe63406d810 0x7fe63406fcd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:22.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe6500b2e90 0x7fe65013e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:22.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe6500a9b00 0x7fe65013dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:22.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 --2- 172.21.15.19:0/3410468440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6500a9720 0x7fe650145760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:22.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 -- 172.21.15.19:0/3410468440 >> 172.21.15.19:0/3410468440 conn(0x7fe650067780 msgr2=0x7fe6500aa2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:22.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 -- 172.21.15.19:0/3410468440 shutdown_connections 2024-05-04T17:45:22.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 -- 172.21.15.19:0/3410468440 wait complete. 2024-05-04T17:45:22.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:22.948+0000 7fe658523640 1 librados: shutdown 2024-05-04T17:45:22.964 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-04T17:45:22.964 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-04T17:45:22.965 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-04T17:45:22.965 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-04T17:45:22.965 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-04T17:45:23.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.066+0000 7f2165074640 10 monclient: get_monmap_and_config 2024-05-04T17:45:23.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.066+0000 7f2165074640 10 monclient: build_initial_monmap 2024-05-04T17:45:23.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.066+0000 7f2165074640 10 monclient: monmap: 2024-05-04T17:45:23.067 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:23.067 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:23.067 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:23.067523+0000 2024-05-04T17:45:23.068 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:23.067523+0000 2024-05-04T17:45:23.068 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:23.068 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:23.068 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:23.068 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:23.068 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:23.068 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:23.068 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:23.068 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:23.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.067+0000 7f2165074640 1 Processor -- start 2024-05-04T17:45:23.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.067+0000 7f2165074640 1 -- start start 2024-05-04T17:45:23.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.067+0000 7f2165074640 10 monclient: init 2024-05-04T17:45:23.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:23.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-05-04T17:45:23.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21600a9fb0 0x7f21600aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:23.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 10 monclient(hunting): picked mon.noname-b con 0x7f21600a9fb0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:23.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f21600b27f0 0x7f216013fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 10 monclient(hunting): picked mon.noname-c con 0x7f21600b27f0 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2160140310 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21600ab880 con 0x7f2160140310 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f21601407a0 con 0x7f21600b27f0 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f2160146b00 con 0x7f21600a9fb0 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f2165074640 10 monclient(hunting): authenticate will time out at 1345.181778s 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.068+0000 7f215ed9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21600a9fb0 0x7f21600aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f215ed9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21600a9fb0 0x7f21600aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:36874/0 (socket says 172.21.15.19:36874) 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f215ed9d640 1 -- 172.21.15.19:0/2689375802 learned_addr learned my addr 172.21.15.19:0/2689375802 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f215ed9d640 10 monclient(hunting): get_auth_request con 0x7f21600a9fb0 auth_method 0 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f215ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f215ed9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:23.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f215ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:23.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f2157fff640 1 --2- 172.21.15.19:0/2689375802 >> v2:172.21.15.186:3300/0 conn(0x7f21600b27f0 0x7f216013fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:23.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f2157fff640 10 monclient(hunting): get_auth_request con 0x7f21600b27f0 auth_method 0 2024-05-04T17:45:23.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f2157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:23.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f2157fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:23.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f2157fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:23.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f215ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:23.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f215ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:23.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.069+0000 7f215ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:23.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.070+0000 7f2157fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:23.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.070+0000 7f2157fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:23.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.070+0000 7f2157fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:23.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.071+0000 7f215ed9d640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-05-04T17:45:23.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.071+0000 7f215ed9d640 1 -- 172.21.15.19:0/2689375802 >> v2:172.21.15.186:3300/0 conn(0x7f21600b27f0 msgr2=0x7f216013fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:23.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.071+0000 7f215ed9d640 1 --2- 172.21.15.19:0/2689375802 >> v2:172.21.15.186:3300/0 conn(0x7f21600b27f0 0x7f216013fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:23.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.071+0000 7f215ed9d640 1 -- 172.21.15.19:0/2689375802 >> v1:172.21.15.186:6789/0 conn(0x7f2160140310 legacy=0x7f21601463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:23.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.071+0000 7f215ed9d640 10 monclient: _finish_hunting 0 2024-05-04T17:45:23.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.071+0000 7f215ed9d640 1 monclient: found mon.noname-b 2024-05-04T17:45:23.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.071+0000 7f215ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:23.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.071+0000 7f215ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:45:23.073 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.071+0000 7f215ed9d640 1 -- 172.21.15.19:0/2689375802 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2160146cb0 con 0x7f21600a9fb0 2024-05-04T17:45:23.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.073+0000 7f215ed9d640 1 --2- 172.21.15.19:0/2689375802 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21600a9fb0 0x7f21600aa390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2150009870 tx=0x7f215002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bf7995f690feb19 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:23.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 1 -- 172.21.15.19:0/2689375802 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150035020 con 0x7f21600a9fb0 2024-05-04T17:45:23.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:23.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:23.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 10 monclient: dump: 2024-05-04T17:45:23.078 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:23.078 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:23.078 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:23.078 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:23.078 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:23.078 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:23.078 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 10 monclient: _finish_auth 0 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 10 monclient: _check_auth_tickets 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 1 -- 172.21.15.19:0/2689375802 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2150031640 con 0x7f21600a9fb0 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f2165074640 5 monclient: authenticate success, global_id 4241 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f2165074640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f2165074640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 1 -- 172.21.15.19:0/2689375802 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215002d040 con 0x7f21600a9fb0 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f2165074640 4 set_mon_vals no callback set 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:23.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:23.080 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.077+0000 7f215e59c640 10 monclient: dump: 2024-05-04T17:45:23.080 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:23.080 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:23.080 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:23.080 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:23.080 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:23.080 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:23.080 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:23.080 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:23.080 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:23.080 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:23.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.078+0000 7f2165074640 10 monclient: shutdown 2024-05-04T17:45:23.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.078+0000 7f2165074640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:23.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.078+0000 7f2165074640 1 -- 172.21.15.19:0/2689375802 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21600a9fb0 msgr2=0x7f21600aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:23.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.078+0000 7f2165074640 1 --2- 172.21.15.19:0/2689375802 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21600a9fb0 0x7f21600aa390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2150009870 tx=0x7f215002b8f0 comp rx=0 tx=0).stop 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.078+0000 7f2165074640 1 -- 172.21.15.19:0/2689375802 shutdown_connections 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.078+0000 7f2165074640 1 --2- 172.21.15.19:0/2689375802 >> v2:172.21.15.186:3300/0 conn(0x7f21600b27f0 0x7f216013fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.078+0000 7f2165074640 1 --2- 172.21.15.19:0/2689375802 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21600a9fb0 0x7f21600aa390 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.078+0000 7f2165074640 1 -- 172.21.15.19:0/2689375802 >> 172.21.15.19:0/2689375802 conn(0x7f21600b4e90 msgr2=0x7f21600b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.078+0000 7f2165074640 1 -- 172.21.15.19:0/2689375802 shutdown_connections 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.078+0000 7f2165074640 1 -- 172.21.15.19:0/2689375802 wait complete. 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.079+0000 7f2165074640 10 monclient: build_initial_monmap 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.079+0000 7f2165074640 10 monclient: monmap: 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:23.084 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.079+0000 7f2165074640 1 librados: starting msgr at 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.079+0000 7f2165074640 1 librados: starting objecter 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.079+0000 7f2165074640 1 Processor -- start 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.079+0000 7f2165074640 1 -- start start 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.079+0000 7f2165074640 1 librados: setting wanted keys 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.079+0000 7f2165074640 1 librados: calling monclient init 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.079+0000 7f2165074640 10 monclient: init 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21600a9fb0 0x7f21600b8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 10 monclient(hunting): picked mon.noname-b con 0x7f21600a9fb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21600b27f0 0x7f2160139c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:23.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 10 monclient(hunting): picked mon.noname-c con 0x7f21600b27f0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2160140310 0x7f216013a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 10 monclient(hunting): picked mon.noname-a con 0x7f2160140310 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f216014d7a0 con 0x7f2160140310 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f216014d620 con 0x7f21600a9fb0 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f216014d920 con 0x7f21600b27f0 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2165074640 10 monclient(hunting): authenticate will time out at 1345.193976s 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f2157fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21600b27f0 0x7f2160139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f215ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21600a9fb0 0x7f21600b8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.080+0000 7f215f59e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2160140310 0x7f216013a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21600a9fb0 0x7f21600b8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36212/0 (socket says 172.21.15.19:36212) 2024-05-04T17:45:23.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215ed9d640 1 -- 172.21.15.19:0/85160739 learned_addr learned my addr 172.21.15.19:0/85160739 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215ed9d640 10 monclient(hunting): get_auth_request con 0x7f21600a9fb0 auth_method 0 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215ed9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f2157fff640 10 monclient(hunting): get_auth_request con 0x7f21600b27f0 auth_method 0 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f2157fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f2157fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f2157fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215f59e640 10 monclient(hunting): get_auth_request con 0x7f2160140310 auth_method 0 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215f59e640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215f59e640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:23.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f215ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:23.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f2157fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:23.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f2157fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:23.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.081+0000 7f2157fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:23.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.082+0000 7f215f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:23.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.082+0000 7f215f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:23.088 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.082+0000 7f215f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.084+0000 7f215ed9d640 10 monclient(hunting): handle_auth_done global_id 4315 payload 824 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.084+0000 7f215ed9d640 1 -- 172.21.15.19:0/85160739 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21600b27f0 msgr2=0x7f2160139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.084+0000 7f215ed9d640 1 --2- 172.21.15.19:0/85160739 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21600b27f0 0x7f2160139c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.084+0000 7f215ed9d640 1 -- 172.21.15.19:0/85160739 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2160140310 msgr2=0x7f216013a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.084+0000 7f215ed9d640 1 --2- 172.21.15.19:0/85160739 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2160140310 0x7f216013a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.084+0000 7f215ed9d640 10 monclient: _finish_hunting 0 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.084+0000 7f215ed9d640 1 monclient: found mon.noname-b 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.084+0000 7f215ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.084+0000 7f215ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.084+0000 7f215ed9d640 1 -- 172.21.15.19:0/85160739 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f216013ad60 con 0x7f21600a9fb0 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.086+0000 7f215ed9d640 1 --2- 172.21.15.19:0/85160739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21600a9fb0 0x7f21600b8180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f215002b930 tx=0x7f21500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.086+0000 7f21577fe640 10 client.?.objecter ms_handle_connect 0x7f21600a9fb0 2024-05-04T17:45:23.089 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.086+0000 7f21577fe640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:23.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 1 -- 172.21.15.19:0/85160739 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150040070 con 0x7f21600a9fb0 2024-05-04T17:45:23.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:23.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:23.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 10 monclient: dump: 2024-05-04T17:45:23.092 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:23.092 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:23.092 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:23.092 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:23.092 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:23.092 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:23.092 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:23.092 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f2165074640 5 monclient: authenticate success, global_id 4315 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 1 -- 172.21.15.19:0/85160739 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f215003e070 con 0x7f21600a9fb0 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 1 -- 172.21.15.19:0/85160739 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150035040 con 0x7f21600a9fb0 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f2165074640 10 monclient: _renew_subs 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f2165074640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f2165074640 1 -- 172.21.15.19:0/85160739 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21601db550 con 0x7f21600a9fb0 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f215dd9b640 4 set_mon_vals no callback set 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:23.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f21577fe640 10 monclient: dump: 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f2165074640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f2165074640 10 monclient: _renew_subs 2024-05-04T17:45:23.094 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f2165074640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:23.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f2165074640 1 -- 172.21.15.19:0/85160739 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21601dbaf0 con 0x7f21600a9fb0 2024-05-04T17:45:23.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.091+0000 7f2165074640 1 librados: init done 2024-05-04T17:45:23.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.092+0000 7f2165074640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:23.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.092+0000 7f2165074640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:23.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.092+0000 7f2165074640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:23.095 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.092+0000 7f2165074640 1 -- 172.21.15.19:0/85160739 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2160068b10 con 0x7f21600a9fb0 2024-05-04T17:45:23.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.113+0000 7f21577fe640 1 -- 172.21.15.19:0/85160739 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2150038020 con 0x7f21600a9fb0 2024-05-04T17:45:23.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.114+0000 7f21577fe640 1 --2- 172.21.15.19:0/85160739 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f214006d7c0 0x7f214006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:23.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.114+0000 7f2157fff640 1 --2- 172.21.15.19:0/85160739 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f214006d7c0 0x7f214006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:23.114 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.114+0000 7f2157fff640 10 monclient: get_auth_request con 0x7f214006d7c0 auth_method 0 2024-05-04T17:45:23.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.114+0000 7f2157fff640 1 --2- 172.21.15.19:0/85160739 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f214006d7c0 0x7f214006fc80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f21600b87e0 tx=0x7f2148009190 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:23.115 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.114+0000 7f21577fe640 10 client.4315.objecter ms_handle_connect 0x7f214006d7c0 2024-05-04T17:45:23.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.440+0000 7f21577fe640 1 -- 172.21.15.19:0/85160739 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5210+0+0 (secure 0 0 0) 0x7f2150078db0 con 0x7f21600a9fb0 2024-05-04T17:45:23.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.440+0000 7f21577fe640 10 client.4315.objecter ms_dispatch 0x7f2160000b90 osd_map(24..24 src has 1..24) 2024-05-04T17:45:23.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.440+0000 7f21577fe640 3 client.4315.objecter handle_osd_map got epochs [24,24] > 0 2024-05-04T17:45:23.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.440+0000 7f21577fe640 3 client.4315.objecter handle_osd_map decoding full epoch 24 2024-05-04T17:45:23.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.440+0000 7f21577fe640 20 client.4315.objecter dump_active .. 0 homeless 2024-05-04T17:45:23.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.451+0000 7f21577fe640 1 -- 172.21.15.19:0/85160739 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f215007ad10 con 0x7f21600a9fb0 2024-05-04T17:45:23.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.451+0000 7f21577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:23.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.451+0000 7f21577fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:23.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.706+0000 7f2165074640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-05-04T17:45:23.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.706+0000 7f2165074640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-05-04T17:45:23.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.706+0000 7f2165074640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:23.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.706+0000 7f2165074640 1 -- 172.21.15.19:0/85160739 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f216013f310 con 0x7f21600a9fb0 2024-05-04T17:45:23.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.853+0000 7f21577fe640 1 -- 172.21.15.19:0/85160739 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+172941 (secure 0 0 0) 0x7f215007f010 con 0x7f21600a9fb0 2024-05-04T17:45:23.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.853+0000 7f21577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-05-04T17:45:23.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.853+0000 7f21577fe640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:23.857 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:45:23.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.855+0000 7f2165074640 10 librados: watch_flush enter 2024-05-04T17:45:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 10 librados: watch_flush exit 2024-05-04T17:45:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 20 client.4315.objecter shutdown clearing up homeless session... 2024-05-04T17:45:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 10 client.4315.objecter successfully canceled tick 2024-05-04T17:45:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 -- 172.21.15.19:0/85160739 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f214006d7c0 msgr2=0x7f214006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 --2- 172.21.15.19:0/85160739 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f214006d7c0 0x7f214006fc80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f21600b87e0 tx=0x7f2148009190 comp rx=0 tx=0).stop 2024-05-04T17:45:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 10 monclient: shutdown 2024-05-04T17:45:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 -- 172.21.15.19:0/85160739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21600a9fb0 msgr2=0x7f21600b8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 --2- 172.21.15.19:0/85160739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21600a9fb0 0x7f21600b8180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f215002b930 tx=0x7f21500046c0 comp rx=0 tx=0).stop 2024-05-04T17:45:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 -- 172.21.15.19:0/85160739 shutdown_connections 2024-05-04T17:45:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 --2- 172.21.15.19:0/85160739 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f214006d7c0 0x7f214006fc80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:23.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 --2- 172.21.15.19:0/85160739 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2160140310 0x7f216013a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:23.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 --2- 172.21.15.19:0/85160739 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21600b27f0 0x7f2160139c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:23.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 --2- 172.21.15.19:0/85160739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21600a9fb0 0x7f21600b8180 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:23.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 -- 172.21.15.19:0/85160739 >> 172.21.15.19:0/85160739 conn(0x7f21600b4e90 msgr2=0x7f21600b5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:23.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 -- 172.21.15.19:0/85160739 shutdown_connections 2024-05-04T17:45:23.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 -- 172.21.15.19:0/85160739 wait complete. 2024-05-04T17:45:23.861 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.856+0000 7f2165074640 1 librados: shutdown 2024-05-04T17:45:23.867 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":5,"flags":0,"active_gid":4117,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":327322581},{"type":"v1","addr":"172.21.15.186:6801","nonce":327322581}]},"active_addr":"172.21.15.186:6801/327322581","active_change":"2024-05-04T17:44:22.537124+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4114,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":2577147644}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":1479219661}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":864235397}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":286608058}]}]} 2024-05-04T17:45:23.874 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-04T17:45:23.874 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-04T17:45:23.874 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-04T17:45:23.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.978+0000 7fee29cc8640 10 monclient: get_monmap_and_config 2024-05-04T17:45:23.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.978+0000 7fee29cc8640 10 monclient: build_initial_monmap 2024-05-04T17:45:23.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.978+0000 7fee29cc8640 10 monclient: monmap: 2024-05-04T17:45:23.979 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:23.979 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:23.979 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:23.979352+0000 2024-05-04T17:45:23.979 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:23.979352+0000 2024-05-04T17:45:23.979 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:23.979 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:23.979 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:23.980 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:23.980 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:23.980 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:23.980 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:23.980 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:23.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.979+0000 7fee29cc8640 1 Processor -- start 2024-05-04T17:45:23.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.979+0000 7fee29cc8640 1 -- start start 2024-05-04T17:45:23.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.979+0000 7fee29cc8640 10 monclient: init 2024-05-04T17:45:23.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.979+0000 7fee29cc8640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:23.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.979+0000 7fee29cc8640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.979+0000 7fee29cc8640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fee240a9fb0 0x7fee240aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.979+0000 7fee29cc8640 10 monclient(hunting): picked mon.noname-b con 0x7fee240a9fb0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.979+0000 7fee29cc8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee240b27f0 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.980+0000 7fee29cc8640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fee241404e0 0x7fee241463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.980+0000 7fee29cc8640 10 monclient(hunting): picked mon.noname-a con 0x7fee241404e0 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.980+0000 7fee29cc8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee240ab880 con 0x7fee240b27f0 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.980+0000 7fee29cc8640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.980+0000 7fee29cc8640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fee24146930 con 0x7fee241404e0 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.980+0000 7fee29cc8640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.980+0000 7fee29cc8640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fee24146ab0 con 0x7fee240a9fb0 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.980+0000 7fee29cc8640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.980+0000 7fee29cc8640 10 monclient(hunting): authenticate will time out at 1346.093695s 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.980+0000 7fee23fff640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fee241404e0 0x7fee241463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:23.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.981+0000 7fee23fff640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fee241404e0 0x7fee241463f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37128/0 (socket says 172.21.15.19:37128) 2024-05-04T17:45:23.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.981+0000 7fee23fff640 1 -- 172.21.15.19:0/2499138901 learned_addr learned my addr 172.21.15.19:0/2499138901 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:23.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.981+0000 7fee23fff640 10 monclient(hunting): get_auth_request con 0x7fee241404e0 auth_method 0 2024-05-04T17:45:23.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.981+0000 7fee23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:23.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.981+0000 7fee23fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:23.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.981+0000 7fee23fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:23.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.982+0000 7fee23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:23.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.982+0000 7fee23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:23.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.982+0000 7fee23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:23.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.984+0000 7fee23fff640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2024-05-04T17:45:23.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.984+0000 7fee23fff640 1 -- 172.21.15.19:0/2499138901 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fee240a9fb0 msgr2=0x7fee240aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:23.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.984+0000 7fee23fff640 1 --2- 172.21.15.19:0/2499138901 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fee240a9fb0 0x7fee240aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:23.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.984+0000 7fee23fff640 1 -- 172.21.15.19:0/2499138901 >> v1:172.21.15.19:6789/0 conn(0x7fee240b27f0 legacy=0x7fee2413fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:23.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.984+0000 7fee23fff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:23.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.984+0000 7fee23fff640 1 monclient: found mon.noname-a 2024-05-04T17:45:23.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.984+0000 7fee23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:23.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.984+0000 7fee23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:23.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.984+0000 7fee23fff640 1 -- 172.21.15.19:0/2499138901 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee24146c60 con 0x7fee241404e0 2024-05-04T17:45:23.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.988+0000 7fee23fff640 1 --2- 172.21.15.19:0/2499138901 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee241404e0 0x7fee241463f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009e80 tx=0x7fee1c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9771f74918a809b server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:23.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 1 -- 172.21.15.19:0/2499138901 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c02ca50 con 0x7fee241404e0 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 10 monclient: dump: 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:23.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 10 monclient: _finish_auth 0 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 10 monclient: _check_auth_tickets 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 1 -- 172.21.15.19:0/2499138901 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee1c02cbf0 con 0x7fee241404e0 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee29cc8640 5 monclient: authenticate success, global_id 4272 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee29cc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee29cc8640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 1 -- 172.21.15.19:0/2499138901 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0343f0 con 0x7fee241404e0 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee29cc8640 4 set_mon_vals no callback set 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.996+0000 7fee227fc640 10 monclient: dump: 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:23.998 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:23.999 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:23.999 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:23.999 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:23.999 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:23.999 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:23.999 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:23.999 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:23.999 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:24.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.997+0000 7fee29cc8640 10 monclient: shutdown 2024-05-04T17:45:24.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.997+0000 7fee29cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:24.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.997+0000 7fee29cc8640 1 -- 172.21.15.19:0/2499138901 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee241404e0 msgr2=0x7fee241463f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:24.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.997+0000 7fee29cc8640 1 --2- 172.21.15.19:0/2499138901 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee241404e0 0x7fee241463f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009e80 tx=0x7fee1c02beb0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.997+0000 7fee29cc8640 1 -- 172.21.15.19:0/2499138901 shutdown_connections 2024-05-04T17:45:24.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.997+0000 7fee29cc8640 1 --2- 172.21.15.19:0/2499138901 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee241404e0 0x7fee241463f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.997+0000 7fee29cc8640 1 --2- 172.21.15.19:0/2499138901 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fee240a9fb0 0x7fee240aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.997+0000 7fee29cc8640 1 -- 172.21.15.19:0/2499138901 >> 172.21.15.19:0/2499138901 conn(0x7fee240b4e90 msgr2=0x7fee240b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.997+0000 7fee29cc8640 1 -- 172.21.15.19:0/2499138901 shutdown_connections 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.997+0000 7fee29cc8640 1 -- 172.21.15.19:0/2499138901 wait complete. 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.998+0000 7fee29cc8640 10 monclient: build_initial_monmap 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.998+0000 7fee29cc8640 10 monclient: monmap: 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:24.007 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.998+0000 7fee29cc8640 1 librados: starting msgr at 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.998+0000 7fee29cc8640 1 librados: starting objecter 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.998+0000 7fee29cc8640 1 Processor -- start 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.999+0000 7fee29cc8640 1 -- start start 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.999+0000 7fee29cc8640 1 librados: setting wanted keys 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.999+0000 7fee29cc8640 1 librados: calling monclient init 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.999+0000 7fee29cc8640 10 monclient: init 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.999+0000 7fee29cc8640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.999+0000 7fee29cc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.999+0000 7fee29cc8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee240a9fb0 0x7fee2413a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:23.999+0000 7fee29cc8640 10 monclient(hunting): picked mon.noname-b con 0x7fee240a9fb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee29cc8640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee240b27f0 0x7fee240b8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee29cc8640 10 monclient(hunting): picked mon.noname-a con 0x7fee240b27f0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee29cc8640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fee241404e0 0x7fee240b8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:24.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee29cc8640 10 monclient(hunting): picked mon.noname-c con 0x7fee241404e0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee29cc8640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee29cc8640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fee2414d620 con 0x7fee240b27f0 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee29cc8640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee29cc8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fee24146fe0 con 0x7fee240a9fb0 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee29cc8640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee29cc8640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fee2414d7a0 con 0x7fee241404e0 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee29cc8640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee29cc8640 10 monclient(hunting): authenticate will time out at 1346.113818s 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee237fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee240a9fb0 0x7fee2413a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee237fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee240a9fb0 0x7fee2413a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36214/0 (socket says 172.21.15.19:36214) 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee237fe640 1 -- 172.21.15.19:0/3125455590 learned_addr learned my addr 172.21.15.19:0/3125455590 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee237fe640 10 monclient(hunting): get_auth_request con 0x7fee240a9fb0 auth_method 0 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:24.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee237fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.000+0000 7fee237fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.001+0000 7fee23fff640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fee241404e0 0x7fee240b8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.001+0000 7fee23fff640 10 monclient(hunting): get_auth_request con 0x7fee241404e0 auth_method 0 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.001+0000 7fee23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.001+0000 7fee23fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.001+0000 7fee23fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.001+0000 7fee237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.001+0000 7fee237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.001+0000 7fee237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.002+0000 7fee23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.002+0000 7fee23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.002+0000 7fee23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.005+0000 7fee237fe640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.005+0000 7fee237fe640 1 -- 172.21.15.19:0/3125455590 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fee241404e0 msgr2=0x7fee240b8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:24.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.005+0000 7fee237fe640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fee241404e0 0x7fee240b8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.005+0000 7fee237fe640 1 -- 172.21.15.19:0/3125455590 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee240b27f0 msgr2=0x7fee240b8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:24.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.005+0000 7fee237fe640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee240b27f0 0x7fee240b8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.005+0000 7fee237fe640 10 monclient: _finish_hunting 0 2024-05-04T17:45:24.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.005+0000 7fee237fe640 1 monclient: found mon.noname-b 2024-05-04T17:45:24.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.005+0000 7fee237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:24.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.005+0000 7fee237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:24.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.005+0000 7fee237fe640 1 -- 172.21.15.19:0/3125455590 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee240b8e70 con 0x7fee240a9fb0 2024-05-04T17:45:24.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.005+0000 7fee23fff640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fee241404e0 0x7fee240b8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-04T17:45:24.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.007+0000 7fee237fe640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee240a9fb0 0x7fee2413a8b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fee1400b4d0 tx=0x7fee1400b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:24.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.007+0000 7fee0bfff640 10 client.?.objecter ms_handle_connect 0x7fee240a9fb0 2024-05-04T17:45:24.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.007+0000 7fee0bfff640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:24.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 1 -- 172.21.15.19:0/3125455590 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14014070 con 0x7fee240a9fb0 2024-05-04T17:45:24.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:24.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:24.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 10 monclient: dump: 2024-05-04T17:45:24.013 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:24.013 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 10 monclient: _finish_auth 0 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 10 monclient: _check_auth_tickets 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 1 -- 172.21.15.19:0/3125455590 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee14010870 con 0x7fee240a9fb0 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 1 -- 172.21.15.19:0/3125455590 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14010c60 con 0x7fee240a9fb0 2024-05-04T17:45:24.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee0bfff640 10 monclient: dump: 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee29cc8640 5 monclient: authenticate success, global_id 4318 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee29cc8640 10 monclient: _renew_subs 2024-05-04T17:45:24.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee21ffb640 4 set_mon_vals no callback set 2024-05-04T17:45:24.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee29cc8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:24.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee29cc8640 1 -- 172.21.15.19:0/3125455590 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee24134530 con 0x7fee240a9fb0 2024-05-04T17:45:24.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee29cc8640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:24.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee29cc8640 10 monclient: _renew_subs 2024-05-04T17:45:24.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee29cc8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:24.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee29cc8640 1 -- 172.21.15.19:0/3125455590 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee24134a00 con 0x7fee240a9fb0 2024-05-04T17:45:24.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.012+0000 7fee29cc8640 1 librados: init done 2024-05-04T17:45:24.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.013+0000 7fee29cc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:24.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.013+0000 7fee29cc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:24.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.013+0000 7fee29cc8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:24.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.013+0000 7fee29cc8640 1 -- 172.21.15.19:0/3125455590 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee24068b10 con 0x7fee240a9fb0 2024-05-04T17:45:24.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.033+0000 7fee0bfff640 1 -- 172.21.15.19:0/3125455590 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fee14017070 con 0x7fee240a9fb0 2024-05-04T17:45:24.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.033+0000 7fee0bfff640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fee1006d840 0x7fee1006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:24.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.034+0000 7fee22ffd640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fee1006d840 0x7fee1006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:24.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.034+0000 7fee22ffd640 10 monclient: get_auth_request con 0x7fee1006d840 auth_method 0 2024-05-04T17:45:24.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.034+0000 7fee0bfff640 10 client.4318.objecter ms_handle_connect 0x7fee1006d840 2024-05-04T17:45:24.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.034+0000 7fee22ffd640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fee1006d840 0x7fee1006fd00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fee18000f60 tx=0x7fee180079d0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:24.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.036+0000 7fee0bfff640 1 -- 172.21.15.19:0/3125455590 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5210+0+0 (secure 0 0 0) 0x7fee140573c0 con 0x7fee240a9fb0 2024-05-04T17:45:24.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.036+0000 7fee0bfff640 10 client.4318.objecter ms_dispatch 0x7fee24000b90 osd_map(24..24 src has 1..24) 2024-05-04T17:45:24.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.036+0000 7fee0bfff640 3 client.4318.objecter handle_osd_map got epochs [24,24] > 0 2024-05-04T17:45:24.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.036+0000 7fee0bfff640 3 client.4318.objecter handle_osd_map decoding full epoch 24 2024-05-04T17:45:24.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.036+0000 7fee0bfff640 20 client.4318.objecter dump_active .. 0 homeless 2024-05-04T17:45:24.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.324+0000 7fee0bfff640 1 -- 172.21.15.19:0/3125455590 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fee14059320 con 0x7fee240a9fb0 2024-05-04T17:45:24.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.324+0000 7fee0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:24.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.324+0000 7fee0bfff640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:24.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.527+0000 7fee29cc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:24.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.527+0000 7fee29cc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:24.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.527+0000 7fee29cc8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:24.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.527+0000 7fee29cc8640 1 -- 172.21.15.19:0/3125455590 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fee241db2f0 con 0x7fee240a9fb0 2024-05-04T17:45:24.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.563+0000 7fee0bfff640 1 -- 172.21.15.19:0/3125455590 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+14729 (secure 0 0 0) 0x7fee1405d620 con 0x7fee240a9fb0 2024-05-04T17:45:24.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.563+0000 7fee0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:24.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.563+0000 7fee0bfff640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:24.564 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:45:24.564 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":25,"fsid":"18ced2df-a29f-46cc-b71a-1855bfb60c47","created":"2024-05-04T17:44:16.995866+0000","modified":"2024-05-04T17:45:23.934971+0000","last_up_change":"2024-05-04T17:45:15.621507+0000","last_in_change":"2024-05-04T17:44:23.464900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T17:45:03.247977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-04T17:45:21.183165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b749ec0-22b6-458c-a7af-2ccace806ab7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6809","nonce":2236557705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6811","nonce":2236557705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6815","nonce":2236557705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6813","nonce":2236557705}]},"public_addr":"172.21.15.19:6809/2236557705","cluster_addr":"172.21.15.19:6811/2236557705","heartbeat_back_addr":"172.21.15.19:6815/2236557705","heartbeat_front_addr":"172.21.15.19:6813/2236557705","state":["exists","up"]},{"osd":1,"uuid":"3aa520fe-39c4-4fa2-860b-f4d0cc04ca97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6801","nonce":3360656308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6803","nonce":3360656308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6807","nonce":3360656308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6805","nonce":3360656308}]},"public_addr":"172.21.15.19:6801/3360656308","cluster_addr":"172.21.15.19:6803/3360656308","heartbeat_back_addr":"172.21.15.19:6807/3360656308","heartbeat_front_addr":"172.21.15.19:6805/3360656308","state":["exists","up"]},{"osd":2,"uuid":"16ed21c9-e5c3-4d65-b4d1-e4927a0fe919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6817","nonce":3744614279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6820","nonce":3744614279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6825","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6827","nonce":3744614279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6823","nonce":3744614279}]},"public_addr":"172.21.15.19:6817/3744614279","cluster_addr":"172.21.15.19:6820/3744614279","heartbeat_back_addr":"172.21.15.19:6827/3744614279","heartbeat_front_addr":"172.21.15.19:6823/3744614279","state":["exists","up"]},{"osd":3,"uuid":"45aa2c0f-cfae-478c-a60c-4c1491e45525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6819","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6821","nonce":2537429684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6826","nonce":2537429684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6831","nonce":2537429684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6829","nonce":2537429684}]},"public_addr":"172.21.15.19:6821/2537429684","cluster_addr":"172.21.15.19:6826/2537429684","heartbeat_back_addr":"172.21.15.19:6831/2537429684","heartbeat_front_addr":"172.21.15.19:6829/2537429684","state":["exists","up"]},{"osd":4,"uuid":"cb00f84f-d78c-461c-9843-73a9da4b80a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6819","nonce":356995042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6821","nonce":356995042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6825","nonce":356995042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6823","nonce":356995042}]},"public_addr":"172.21.15.186:6819/356995042","cluster_addr":"172.21.15.186:6821/356995042","heartbeat_back_addr":"172.21.15.186:6825/356995042","heartbeat_front_addr":"172.21.15.186:6823/356995042","state":["exists","up"]},{"osd":5,"uuid":"d07ad403-291f-4bde-a64a-5181fb79e544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6811","nonce":3728813703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6813","nonce":3728813703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6817","nonce":3728813703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6815","nonce":3728813703}]},"public_addr":"172.21.15.186:6811/3728813703","cluster_addr":"172.21.15.186:6813/3728813703","heartbeat_back_addr":"172.21.15.186:6817/3728813703","heartbeat_front_addr":"172.21.15.186:6815/3728813703","state":["exists","up"]},{"osd":6,"uuid":"5110d1ad-5634-4c8e-9338-d6c4229b1d63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6803","nonce":4252462313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6805","nonce":4252462313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6809","nonce":4252462313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6807","nonce":4252462313}]},"public_addr":"172.21.15.186:6803/4252462313","cluster_addr":"172.21.15.186:6805/4252462313","heartbeat_back_addr":"172.21.15.186:6809/4252462313","heartbeat_front_addr":"172.21.15.186:6807/4252462313","state":["exists","up"]},{"osd":7,"uuid":"b307b905-5cc8-447b-8765-d888ed6f655b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6827","nonce":1403900459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6829","nonce":1403900459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6833","nonce":1403900459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6831","nonce":1403900459}]},"public_addr":"172.21.15.186:6827/1403900459","cluster_addr":"172.21.15.186:6829/1403900459","heartbeat_back_addr":"172.21.15.186:6833/1403900459","heartbeat_front_addr":"172.21.15.186:6831/1403900459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.360122+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.570945+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:01.075527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.428399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.550872+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:44:59.984305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.443005+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.439608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:45:24.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.564+0000 7fee29cc8640 10 librados: watch_flush enter 2024-05-04T17:45:24.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.564+0000 7fee29cc8640 10 librados: watch_flush exit 2024-05-04T17:45:24.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.564+0000 7fee29cc8640 20 client.4318.objecter shutdown clearing up homeless session... 2024-05-04T17:45:24.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.564+0000 7fee29cc8640 10 client.4318.objecter successfully canceled tick 2024-05-04T17:45:24.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 -- 172.21.15.19:0/3125455590 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fee1006d840 msgr2=0x7fee1006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:24.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fee1006d840 0x7fee1006fd00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fee18000f60 tx=0x7fee180079d0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 10 monclient: shutdown 2024-05-04T17:45:24.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:24.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 -- 172.21.15.19:0/3125455590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee240a9fb0 msgr2=0x7fee2413a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:24.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee240a9fb0 0x7fee2413a8b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fee1400b4d0 tx=0x7fee1400b990 comp rx=0 tx=0).stop 2024-05-04T17:45:24.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 -- 172.21.15.19:0/3125455590 shutdown_connections 2024-05-04T17:45:24.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fee1006d840 0x7fee1006fd00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fee241404e0 0x7fee240b8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fee240b27f0 0x7fee240b8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 --2- 172.21.15.19:0/3125455590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee240a9fb0 0x7fee2413a8b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 -- 172.21.15.19:0/3125455590 >> 172.21.15.19:0/3125455590 conn(0x7fee240b4e90 msgr2=0x7fee2413bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:24.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 -- 172.21.15.19:0/3125455590 shutdown_connections 2024-05-04T17:45:24.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 -- 172.21.15.19:0/3125455590 wait complete. 2024-05-04T17:45:24.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.565+0000 7fee29cc8640 1 librados: shutdown 2024-05-04T17:45:24.577 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-04T17:45:24.577 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-04T17:45:24.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.684+0000 7fe510a55640 10 monclient: get_monmap_and_config 2024-05-04T17:45:24.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.684+0000 7fe510a55640 10 monclient: build_initial_monmap 2024-05-04T17:45:24.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.684+0000 7fe510a55640 10 monclient: monmap: 2024-05-04T17:45:24.685 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:24.685 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:24.685 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:24.685466+0000 2024-05-04T17:45:24.686 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:24.685466+0000 2024-05-04T17:45:24.686 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:24.686 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:24.686 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:24.686 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:24.686 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:24.686 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:24.686 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:24.686 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:24.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.685+0000 7fe510a55640 1 Processor -- start 2024-05-04T17:45:24.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.685+0000 7fe510a55640 1 -- start start 2024-05-04T17:45:24.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.685+0000 7fe510a55640 10 monclient: init 2024-05-04T17:45:24.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.685+0000 7fe510a55640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:24.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.685+0000 7fe510a55640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-05-04T17:45:24.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.685+0000 7fe510a55640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fe50c0a9720 0x7fe50c0aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:24.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.685+0000 7fe510a55640 10 monclient(hunting): picked mon.noname-a con 0x7fe50c0a9720 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:24.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.685+0000 7fe510a55640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe50c0a9b00 0x7fe50c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.685+0000 7fe510a55640 10 monclient(hunting): picked mon.noname-b con 0x7fe50c0a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.685+0000 7fe510a55640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe50c146890 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.685+0000 7fe510a55640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe50c0abc40 con 0x7fe50c146890 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.685+0000 7fe510a55640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.686+0000 7fe510a55640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fe50c146090 con 0x7fe50c0a9720 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.686+0000 7fe510a55640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.686+0000 7fe510a55640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fe50c146370 con 0x7fe50c0a9b00 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.686+0000 7fe510a55640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.686+0000 7fe510a55640 10 monclient(hunting): authenticate will time out at 1346.799648s 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.686+0000 7fe50a59c640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fe50c0a9720 0x7fe50c0aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.686+0000 7fe50ad9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe50c146890 0x7fe50c13fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:46420/0 (socket says 172.21.15.19:46420) 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.686+0000 7fe50ad9d640 1 -- 172.21.15.19:0/2868801447 learned_addr learned my addr 172.21.15.19:0/2868801447 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.686+0000 7fe501d9b640 1 --2- 172.21.15.19:0/2868801447 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe50c0a9b00 0x7fe50c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe50a59c640 10 monclient(hunting): get_auth_request con 0x7fe50c0a9720 auth_method 0 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe50a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:24.689 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe50a59c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:24.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe50a59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:24.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe501d9b640 10 monclient(hunting): get_auth_request con 0x7fe50c0a9b00 auth_method 0 2024-05-04T17:45:24.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe501d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:24.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe501d9b640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:24.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe501d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:24.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe50a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:24.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe50a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:24.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe50a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:24.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe501d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:24.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe501d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:24.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.687+0000 7fe501d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:24.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe509d9b640 1 -- 172.21.15.19:0/2868801447 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3697549186 0 0) 0x7fe50c0abc40 con 0x7fe50c146890 2024-05-04T17:45:24.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe509d9b640 10 monclient(hunting): _negotiate 2024-05-04T17:45:24.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe509d9b640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:24.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe509d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:24.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe509d9b640 10 monclient(hunting): my global_id is 4321 2024-05-04T17:45:24.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe509d9b640 1 -- 172.21.15.19:0/2868801447 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4f8003520 con 0x7fe50c146890 2024-05-04T17:45:24.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe501d9b640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-05-04T17:45:24.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe501d9b640 1 -- 172.21.15.19:0/2868801447 >> v2:172.21.15.19:3300/0 conn(0x7fe50c0a9720 msgr2=0x7fe50c0aa890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:24.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe501d9b640 1 --2- 172.21.15.19:0/2868801447 >> v2:172.21.15.19:3300/0 conn(0x7fe50c0a9720 0x7fe50c0aa890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe501d9b640 1 -- 172.21.15.19:0/2868801447 >> v1:172.21.15.186:6789/0 conn(0x7fe50c146890 legacy=0x7fe50c13fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:24.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe501d9b640 10 monclient: _finish_hunting 0 2024-05-04T17:45:24.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe501d9b640 1 monclient: found mon.noname-b 2024-05-04T17:45:24.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe501d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:24.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe501d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:45:24.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe501d9b640 1 -- 172.21.15.19:0/2868801447 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50c1405b0 con 0x7fe50c0a9b00 2024-05-04T17:45:24.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.689+0000 7fe50a59c640 1 --2- 172.21.15.19:0/2868801447 >> v2:172.21.15.19:3300/0 conn(0x7fe50c0a9720 0x7fe50c0aa890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-04T17:45:24.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.691+0000 7fe501d9b640 1 --2- 172.21.15.19:0/2868801447 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe50c0a9b00 0x7fe50c0b2e90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe4f40097a0 tx=0x7fe4f402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab906bcc7eeb4a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:24.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 1 -- 172.21.15.19:0/2868801447 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4035020 con 0x7fe50c0a9b00 2024-05-04T17:45:24.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:24.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:24.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 10 monclient: dump: 2024-05-04T17:45:24.696 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:24.696 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:24.696 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:24.696 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 10 monclient: _finish_auth 0 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 10 monclient: _check_auth_tickets 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 1 -- 172.21.15.19:0/2868801447 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f4031830 con 0x7fe50c0a9b00 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe510a55640 5 monclient: authenticate success, global_id 4250 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 1 -- 172.21.15.19:0/2868801447 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4031c00 con 0x7fe50c0a9b00 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:24.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe509d9b640 10 monclient: dump: 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe510a55640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:24.698 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.695+0000 7fe510a55640 4 set_mon_vals no callback set 2024-05-04T17:45:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.696+0000 7fe510a55640 10 monclient: shutdown 2024-05-04T17:45:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.696+0000 7fe510a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.696+0000 7fe510a55640 1 -- 172.21.15.19:0/2868801447 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe50c0a9b00 msgr2=0x7fe50c0b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.696+0000 7fe510a55640 1 --2- 172.21.15.19:0/2868801447 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe50c0a9b00 0x7fe50c0b2e90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe4f40097a0 tx=0x7fe4f402b8d0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.696+0000 7fe510a55640 1 -- 172.21.15.19:0/2868801447 shutdown_connections 2024-05-04T17:45:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.696+0000 7fe510a55640 1 --2- 172.21.15.19:0/2868801447 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe50c0a9b00 0x7fe50c0b2e90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.696+0000 7fe510a55640 1 --2- 172.21.15.19:0/2868801447 >> v2:172.21.15.19:3300/0 conn(0x7fe50c0a9720 0x7fe50c0aa890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.696+0000 7fe510a55640 1 -- 172.21.15.19:0/2868801447 >> 172.21.15.19:0/2868801447 conn(0x7fe50c067780 msgr2=0x7fe50c0b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.696+0000 7fe510a55640 1 -- 172.21.15.19:0/2868801447 shutdown_connections 2024-05-04T17:45:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.696+0000 7fe510a55640 1 -- 172.21.15.19:0/2868801447 wait complete. 2024-05-04T17:45:24.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.697+0000 7fe510a55640 10 monclient: build_initial_monmap 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.697+0000 7fe510a55640 10 monclient: monmap: 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.698+0000 7fe510a55640 1 librados: starting msgr at 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.698+0000 7fe510a55640 1 librados: starting objecter 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.698+0000 7fe510a55640 1 Processor -- start 2024-05-04T17:45:24.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.698+0000 7fe510a55640 1 -- start start 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.698+0000 7fe510a55640 1 librados: setting wanted keys 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.698+0000 7fe510a55640 1 librados: calling monclient init 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.698+0000 7fe510a55640 10 monclient: init 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe50c0a9720 0x7fe50c148420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 10 monclient(hunting): picked mon.noname-a con 0x7fe50c0a9720 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe50c0a9b00 0x7fe50c1d85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 10 monclient(hunting): picked mon.noname-b con 0x7fe50c0a9b00 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe50c146890 0x7fe50c14c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 10 monclient(hunting): picked mon.noname-c con 0x7fe50c146890 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c14d420 con 0x7fe50c0a9720 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:24.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c146f70 con 0x7fe50c0a9b00 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fe50c14d5a0 con 0x7fe50c146890 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.699+0000 7fe510a55640 10 monclient(hunting): authenticate will time out at 1346.813394s 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe50ad9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe50c146890 0x7fe50c14c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe50a59c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe50c0a9720 0x7fe50c148420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe50ad9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe50c146890 0x7fe50c14c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:36932/0 (socket says 172.21.15.19:36932) 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe50ad9d640 1 -- 172.21.15.19:0/874447984 learned_addr learned my addr 172.21.15.19:0/874447984 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe50ad9d640 10 monclient(hunting): get_auth_request con 0x7fe50c146890 auth_method 0 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe50ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe50ad9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe50ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe501d9b640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe50c0a9b00 0x7fe50c1d85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe501d9b640 10 monclient(hunting): get_auth_request con 0x7fe50c0a9b00 auth_method 0 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe501d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:24.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe501d9b640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe501d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe50a59c640 10 monclient(hunting): get_auth_request con 0x7fe50c0a9720 auth_method 0 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe50a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe50a59c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.700+0000 7fe50a59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.701+0000 7fe50ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.701+0000 7fe50ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.701+0000 7fe50ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.701+0000 7fe501d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.701+0000 7fe501d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.701+0000 7fe501d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.701+0000 7fe50a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:24.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.701+0000 7fe50a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:24.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.701+0000 7fe50a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:24.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.704+0000 7fe501d9b640 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2024-05-04T17:45:24.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.704+0000 7fe501d9b640 1 -- 172.21.15.19:0/874447984 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe50c146890 msgr2=0x7fe50c14c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:24.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.704+0000 7fe501d9b640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe50c146890 0x7fe50c14c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.704+0000 7fe501d9b640 1 -- 172.21.15.19:0/874447984 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe50c0a9720 msgr2=0x7fe50c148420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:24.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.704+0000 7fe501d9b640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe50c0a9720 0x7fe50c148420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:24.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.704+0000 7fe501d9b640 10 monclient: _finish_hunting 0 2024-05-04T17:45:24.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.704+0000 7fe501d9b640 1 monclient: found mon.noname-b 2024-05-04T17:45:24.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.704+0000 7fe501d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:24.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.704+0000 7fe501d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:24.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.704+0000 7fe501d9b640 1 -- 172.21.15.19:0/874447984 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50c1d8080 con 0x7fe50c0a9b00 2024-05-04T17:45:24.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.706+0000 7fe501d9b640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe50c0a9b00 0x7fe50c1d85a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4033000 tx=0x7fe4f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:24.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.706+0000 7fe5037fe640 10 client.?.objecter ms_handle_connect 0x7fe50c0a9b00 2024-05-04T17:45:24.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.707+0000 7fe5037fe640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:24.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 1 -- 172.21.15.19:0/874447984 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4035040 con 0x7fe50c0a9b00 2024-05-04T17:45:24.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:24.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:24.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 10 monclient: dump: 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 1 -- 172.21.15.19:0/874447984 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f403b920 con 0x7fe50c0a9b00 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:24.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 1 -- 172.21.15.19:0/874447984 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f403bd10 con 0x7fe50c0a9b00 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe5037fe640 10 monclient: dump: 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe510a55640 5 monclient: authenticate success, global_id 4324 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe510a55640 10 monclient: _renew_subs 2024-05-04T17:45:24.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe50959a640 4 set_mon_vals no callback set 2024-05-04T17:45:24.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe510a55640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:24.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe510a55640 1 -- 172.21.15.19:0/874447984 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe50c1479d0 con 0x7fe50c0a9b00 2024-05-04T17:45:24.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe510a55640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:24.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe510a55640 10 monclient: _renew_subs 2024-05-04T17:45:24.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe510a55640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:24.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe510a55640 1 -- 172.21.15.19:0/874447984 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe50c148960 con 0x7fe50c0a9b00 2024-05-04T17:45:24.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.710+0000 7fe510a55640 1 librados: init done 2024-05-04T17:45:24.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.711+0000 7fe510a55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:24.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.711+0000 7fe510a55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:24.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.711+0000 7fe510a55640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:24.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.711+0000 7fe510a55640 1 -- 172.21.15.19:0/874447984 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe50c06a700 con 0x7fe50c0a9b00 2024-05-04T17:45:24.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.730+0000 7fe5037fe640 1 -- 172.21.15.19:0/874447984 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe4f4038020 con 0x7fe50c0a9b00 2024-05-04T17:45:24.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.730+0000 7fe5037fe640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe4ec06d810 0x7fe4ec06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:24.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.731+0000 7fe50a59c640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe4ec06d810 0x7fe4ec06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:24.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.731+0000 7fe50a59c640 10 monclient: get_auth_request con 0x7fe4ec06d810 auth_method 0 2024-05-04T17:45:24.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.731+0000 7fe50a59c640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe4ec06d810 0x7fe4ec06fcd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009560 tx=0x7fe4fc009190 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:24.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.731+0000 7fe5037fe640 10 client.4324.objecter ms_handle_connect 0x7fe4ec06d810 2024-05-04T17:45:24.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.963+0000 7fe5037fe640 1 -- 172.21.15.19:0/874447984 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fe4f40317e0 con 0x7fe50c0a9b00 2024-05-04T17:45:24.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.963+0000 7fe5037fe640 10 client.4324.objecter ms_dispatch 0x7fe50c000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:24.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.963+0000 7fe5037fe640 3 client.4324.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:24.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.963+0000 7fe5037fe640 3 client.4324.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:24.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.963+0000 7fe5037fe640 20 client.4324.objecter dump_active .. 0 homeless 2024-05-04T17:45:24.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.976+0000 7fe5037fe640 1 -- 172.21.15.19:0/874447984 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe4f407b510 con 0x7fe50c0a9b00 2024-05-04T17:45:24.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.976+0000 7fe5037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:24.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:24.976+0000 7fe5037fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:25.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.162+0000 7fe510a55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:25.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.162+0000 7fe510a55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:25.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.162+0000 7fe510a55640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.162+0000 7fe510a55640 1 -- 172.21.15.19:0/874447984 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe50c1d8250 con 0x7fe50c0a9b00 2024-05-04T17:45:25.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.189+0000 7fe5037fe640 1 -- 172.21.15.19:0/874447984 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+14729 (secure 0 0 0) 0x7fe4f407f810 con 0x7fe50c0a9b00 2024-05-04T17:45:25.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.189+0000 7fe5037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T17:45:25.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.189+0000 7fe5037fe640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:25.190 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:45:25.190 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":25,"fsid":"18ced2df-a29f-46cc-b71a-1855bfb60c47","created":"2024-05-04T17:44:16.995866+0000","modified":"2024-05-04T17:45:23.934971+0000","last_up_change":"2024-05-04T17:45:15.621507+0000","last_in_change":"2024-05-04T17:44:23.464900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T17:45:03.247977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-04T17:45:21.183165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b749ec0-22b6-458c-a7af-2ccace806ab7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6809","nonce":2236557705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6811","nonce":2236557705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6815","nonce":2236557705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6813","nonce":2236557705}]},"public_addr":"172.21.15.19:6809/2236557705","cluster_addr":"172.21.15.19:6811/2236557705","heartbeat_back_addr":"172.21.15.19:6815/2236557705","heartbeat_front_addr":"172.21.15.19:6813/2236557705","state":["exists","up"]},{"osd":1,"uuid":"3aa520fe-39c4-4fa2-860b-f4d0cc04ca97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6801","nonce":3360656308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6803","nonce":3360656308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6807","nonce":3360656308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6805","nonce":3360656308}]},"public_addr":"172.21.15.19:6801/3360656308","cluster_addr":"172.21.15.19:6803/3360656308","heartbeat_back_addr":"172.21.15.19:6807/3360656308","heartbeat_front_addr":"172.21.15.19:6805/3360656308","state":["exists","up"]},{"osd":2,"uuid":"16ed21c9-e5c3-4d65-b4d1-e4927a0fe919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6817","nonce":3744614279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6820","nonce":3744614279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6825","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6827","nonce":3744614279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6823","nonce":3744614279}]},"public_addr":"172.21.15.19:6817/3744614279","cluster_addr":"172.21.15.19:6820/3744614279","heartbeat_back_addr":"172.21.15.19:6827/3744614279","heartbeat_front_addr":"172.21.15.19:6823/3744614279","state":["exists","up"]},{"osd":3,"uuid":"45aa2c0f-cfae-478c-a60c-4c1491e45525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6819","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6821","nonce":2537429684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6826","nonce":2537429684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6831","nonce":2537429684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6829","nonce":2537429684}]},"public_addr":"172.21.15.19:6821/2537429684","cluster_addr":"172.21.15.19:6826/2537429684","heartbeat_back_addr":"172.21.15.19:6831/2537429684","heartbeat_front_addr":"172.21.15.19:6829/2537429684","state":["exists","up"]},{"osd":4,"uuid":"cb00f84f-d78c-461c-9843-73a9da4b80a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6819","nonce":356995042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6821","nonce":356995042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6825","nonce":356995042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6823","nonce":356995042}]},"public_addr":"172.21.15.186:6819/356995042","cluster_addr":"172.21.15.186:6821/356995042","heartbeat_back_addr":"172.21.15.186:6825/356995042","heartbeat_front_addr":"172.21.15.186:6823/356995042","state":["exists","up"]},{"osd":5,"uuid":"d07ad403-291f-4bde-a64a-5181fb79e544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6811","nonce":3728813703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6813","nonce":3728813703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6817","nonce":3728813703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6815","nonce":3728813703}]},"public_addr":"172.21.15.186:6811/3728813703","cluster_addr":"172.21.15.186:6813/3728813703","heartbeat_back_addr":"172.21.15.186:6817/3728813703","heartbeat_front_addr":"172.21.15.186:6815/3728813703","state":["exists","up"]},{"osd":6,"uuid":"5110d1ad-5634-4c8e-9338-d6c4229b1d63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6803","nonce":4252462313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6805","nonce":4252462313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6809","nonce":4252462313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6807","nonce":4252462313}]},"public_addr":"172.21.15.186:6803/4252462313","cluster_addr":"172.21.15.186:6805/4252462313","heartbeat_back_addr":"172.21.15.186:6809/4252462313","heartbeat_front_addr":"172.21.15.186:6807/4252462313","state":["exists","up"]},{"osd":7,"uuid":"b307b905-5cc8-447b-8765-d888ed6f655b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6827","nonce":1403900459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6829","nonce":1403900459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6833","nonce":1403900459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6831","nonce":1403900459}]},"public_addr":"172.21.15.186:6827/1403900459","cluster_addr":"172.21.15.186:6829/1403900459","heartbeat_back_addr":"172.21.15.186:6833/1403900459","heartbeat_front_addr":"172.21.15.186:6831/1403900459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.360122+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.570945+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:01.075527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.428399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.550872+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:44:59.984305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.443005+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.439608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T17:45:25.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 10 librados: watch_flush enter 2024-05-04T17:45:25.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 10 librados: watch_flush exit 2024-05-04T17:45:25.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 20 client.4324.objecter shutdown clearing up homeless session... 2024-05-04T17:45:25.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 10 client.4324.objecter successfully canceled tick 2024-05-04T17:45:25.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 -- 172.21.15.19:0/874447984 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe4ec06d810 msgr2=0x7fe4ec06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe4ec06d810 0x7fe4ec06fcd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009560 tx=0x7fe4fc009190 comp rx=0 tx=0).stop 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 10 monclient: shutdown 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 -- 172.21.15.19:0/874447984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe50c0a9b00 msgr2=0x7fe50c1d85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe50c0a9b00 0x7fe50c1d85a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4033000 tx=0x7fe4f40046c0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 -- 172.21.15.19:0/874447984 shutdown_connections 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe4ec06d810 0x7fe4ec06fcd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe50c146890 0x7fe50c14c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe50c0a9b00 0x7fe50c1d85a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 --2- 172.21.15.19:0/874447984 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe50c0a9720 0x7fe50c148420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 -- 172.21.15.19:0/874447984 >> 172.21.15.19:0/874447984 conn(0x7fe50c067780 msgr2=0x7fe50c13a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 -- 172.21.15.19:0/874447984 shutdown_connections 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 -- 172.21.15.19:0/874447984 wait complete. 2024-05-04T17:45:25.193 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.191+0000 7fe510a55640 1 librados: shutdown 2024-05-04T17:45:25.203 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-04T17:45:25.204 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-04T17:45:25.204 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-04T17:45:25.204 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-04T17:45:25.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-04T17:45:25.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-04T17:45:25.205 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-04T17:45:25.206 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-04T17:45:25.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.385+0000 7f8306dc7640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.385+0000 7f8306dc7640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.385+0000 7f8306dc7640 10 monclient: monmap: 2024-05-04T17:45:25.386 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.386 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.386 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.386146+0000 2024-05-04T17:45:25.386 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.386146+0000 2024-05-04T17:45:25.387 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.387 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.387 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.387 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.387 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.387 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.387 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.387 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.387+0000 7f8306dc7640 1 Processor -- start 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.387+0000 7f8306dc7640 1 -- start start 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.387+0000 7f8306dc7640 10 monclient: init 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83000a9720 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f83000a9b00 0x7f83000aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 10 monclient(hunting): picked mon.noname-b con 0x7f83000a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f83000b2e90 0x7f830013fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 10 monclient(hunting): picked mon.noname-a con 0x7f83000b2e90 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83000abc40 con 0x7f83000a9720 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f8300146fa0 con 0x7f83000b2e90 2024-05-04T17:45:25.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f8300146090 con 0x7f83000a9b00 2024-05-04T17:45:25.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f8306dc7640 10 monclient(hunting): authenticate will time out at 1347.501974s 2024-05-04T17:45:25.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.389+0000 7f82fffff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f83000a9b00 0x7f83000aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.389+0000 7f82fffff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f83000a9b00 0x7f83000aa890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:36938/0 (socket says 172.21.15.19:36938) 2024-05-04T17:45:25.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.389+0000 7f82fffff640 1 -- 172.21.15.19:0/2551033273 learned_addr learned my addr 172.21.15.19:0/2551033273 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.389+0000 7f82fffff640 10 monclient(hunting): get_auth_request con 0x7f83000a9b00 auth_method 0 2024-05-04T17:45:25.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.389+0000 7f82fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.389+0000 7f82fffff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.389+0000 7f82fffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.391+0000 7f02c9175640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.391+0000 7f02c9175640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.392 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.391+0000 7f02c9175640 10 monclient: monmap: 2024-05-04T17:45:25.392 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.392 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.392 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.392489+0000 2024-05-04T17:45:25.392 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.392489+0000 2024-05-04T17:45:25.392 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.392 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.392 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.392 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.393 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.393 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.393 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.393 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f02c9175640 1 Processor -- start 2024-05-04T17:45:25.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f776b59e640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f776b59e640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.388+0000 7f776b59e640 10 monclient: monmap: 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.389727+0000 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.389727+0000 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.391+0000 7f776b59e640 1 Processor -- start 2024-05-04T17:45:25.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.391+0000 7f776b59e640 1 -- start start 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.391+0000 7f776b59e640 10 monclient: init 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f776b59e640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f776b59e640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f776b59e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f776c06bba0 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f776b59e640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f776c153790 0x7f776c072010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f776b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f776c153790 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f776b59e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f776c072550 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f776b59e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f776c0713c0 con 0x7f776c072550 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f776b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f776c071630 con 0x7f776c06bba0 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f776b59e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f776b59e640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f776c06bf80 con 0x7f776c153790 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f776b59e640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f776b59e640 10 monclient(hunting): authenticate will time out at 1347.506000s 2024-05-04T17:45:25.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f02c9175640 1 -- start start 2024-05-04T17:45:25.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f02c9175640 10 monclient: init 2024-05-04T17:45:25.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-05-04T17:45:25.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f02c40a9720 0x7f02c40aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 10 monclient(hunting): picked mon.noname-b con 0x7f02c40a9720 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02c40a9b00 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:25.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02c40b2e90 0x7f02c413fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 10 monclient(hunting): picked mon.noname-c con 0x7f02c40b2e90 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:25.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02c40abc40 con 0x7f02c40a9b00 2024-05-04T17:45:25.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f02c4146fa0 con 0x7f02c40b2e90 2024-05-04T17:45:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f15c1e3e640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f15c1e3e640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f15c1e3e640 10 monclient: monmap: 2024-05-04T17:45:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.394078+0000 2024-05-04T17:45:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.394078+0000 2024-05-04T17:45:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.398 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.399 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.399 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.399 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f02c4146090 con 0x7f02c40a9720 2024-05-04T17:45:25.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f02c9175640 10 monclient(hunting): authenticate will time out at 1347.507366s 2024-05-04T17:45:25.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.394+0000 7f15c1e3e640 1 Processor -- start 2024-05-04T17:45:25.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.394+0000 7f15c1e3e640 1 -- start start 2024-05-04T17:45:25.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.394+0000 7f15c1e3e640 10 monclient: init 2024-05-04T17:45:25.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.395+0000 7f15c1e3e640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.395+0000 7f15c1e3e640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-05-04T17:45:25.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.395+0000 7f15c1e3e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f15bc072390 0x7f15bc072770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.395+0000 7f15c1e3e640 10 monclient(hunting): picked mon.noname-c con 0x7f15bc072390 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:25.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.395+0000 7f15c1e3e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15bc144180 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:25.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.395+0000 7f15c1e3e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15bc06f540 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.395+0000 7f15c1e3e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15bc071c00 con 0x7f15bc06f540 2024-05-04T17:45:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.395+0000 7f15c1e3e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15bc072d80 con 0x7f15bc144180 2024-05-04T17:45:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.395+0000 7f15c1e3e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.395+0000 7f15c1e3e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f15bc070a60 con 0x7f15bc072390 2024-05-04T17:45:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.395+0000 7f15c1e3e640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.395+0000 7f15c1e3e640 10 monclient(hunting): authenticate will time out at 1347.509155s 2024-05-04T17:45:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.394+0000 7f7a35af2640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f15bb7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f15bc072390 0x7f15bc072770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f15bb7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f15bc072390 0x7f15bc072770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36242/0 (socket says 172.21.15.19:36242) 2024-05-04T17:45:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f15bb7fe640 1 -- 172.21.15.19:0/1625872914 learned_addr learned my addr 172.21.15.19:0/1625872914 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f15bb7fe640 10 monclient(hunting): get_auth_request con 0x7f15bc072390 auth_method 0 2024-05-04T17:45:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f15bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f15bb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f15bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f776a59c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f776c06bba0 0x7f776c153080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:46432/0 (socket says 172.21.15.19:46432) 2024-05-04T17:45:25.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f776a59c640 1 -- 172.21.15.19:0/4282957374 learned_addr learned my addr 172.21.15.19:0/4282957374 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f82fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.394+0000 7f7a35af2640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.394+0000 7f7a35af2640 10 monclient: monmap: 2024-05-04T17:45:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.395775+0000 2024-05-04T17:45:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.395775+0000 2024-05-04T17:45:25.404 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.405 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.405 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.405 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.405 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.405 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.405 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.405 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f02c359e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02c40b2e90 0x7f02c413fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f02c359e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02c40b2e90 0x7f02c413fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36232/0 (socket says 172.21.15.19:36232) 2024-05-04T17:45:25.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f02c359e640 1 -- 172.21.15.19:0/158049333 learned_addr learned my addr 172.21.15.19:0/158049333 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f02c359e640 10 monclient(hunting): get_auth_request con 0x7f02c40b2e90 auth_method 0 2024-05-04T17:45:25.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f02c359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f02c359e640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f02c359e640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.396+0000 7f7a35af2640 1 Processor -- start 2024-05-04T17:45:25.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.396+0000 7f7a35af2640 1 -- start start 2024-05-04T17:45:25.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.396+0000 7f7a35af2640 10 monclient: init 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7a30072390 0x7f7a30072770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 10 monclient(hunting): picked mon.noname-c con 0x7f7a30072390 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f7a30144180 0x7f7a301445a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 10 monclient(hunting): picked mon.noname-a con 0x7f7a30144180 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7a3006f540 0x7f7a3006f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 10 monclient(hunting): picked mon.noname-b con 0x7f7a3006f540 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f7a30071c00 con 0x7f7a30144180 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f7a30072d80 con 0x7f7a30072390 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f7a30070a60 con 0x7f7a3006f540 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.397+0000 7f7a35af2640 10 monclient(hunting): authenticate will time out at 1347.511372s 2024-05-04T17:45:25.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f02c2d9d640 1 --2- 172.21.15.19:0/158049333 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f02c40a9720 0x7f02c40aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.392+0000 7f82fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.393+0000 7f82fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.396+0000 7f82fffff640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2024-05-04T17:45:25.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f15bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f15bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f15bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.396+0000 7f82fffff640 1 -- 172.21.15.19:0/2551033273 >> v2:172.21.15.19:3300/0 conn(0x7f83000b2e90 msgr2=0x7f830013fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.396+0000 7f82fffff640 1 --2- 172.21.15.19:0/2551033273 >> v2:172.21.15.19:3300/0 conn(0x7f83000b2e90 0x7f830013fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.396+0000 7f82fffff640 1 -- 172.21.15.19:0/2551033273 >> v1:172.21.15.19:6789/0 conn(0x7f83000a9720 legacy=0x7f8300146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f02c2d9d640 10 monclient(hunting): get_auth_request con 0x7f02c40a9720 auth_method 0 2024-05-04T17:45:25.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f02c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f02c2d9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f02c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.396+0000 7f82fffff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.396+0000 7f82fffff640 1 monclient: found mon.noname-b 2024-05-04T17:45:25.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.396+0000 7f82fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.396+0000 7f82fffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f21cc570640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f21cc570640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f21cc570640 10 monclient: monmap: 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.399397+0000 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.399397+0000 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.412 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.396+0000 7f82fffff640 1 -- 172.21.15.19:0/2551033273 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8300146370 con 0x7f83000a9b00 2024-05-04T17:45:25.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f6739d84640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f6739d84640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.398+0000 7f6739d84640 10 monclient: monmap: 2024-05-04T17:45:25.413 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.413 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.413 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.399732+0000 2024-05-04T17:45:25.414 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.399732+0000 2024-05-04T17:45:25.414 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.414 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.414 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.414 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.414 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.414 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.414 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.414 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.399+0000 7f02c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.399+0000 7f02c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.399+0000 7f02c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.399+0000 7f21cc570640 1 Processor -- start 2024-05-04T17:45:25.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.399+0000 7f6739d84640 1 Processor -- start 2024-05-04T17:45:25.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f6739d84640 1 -- start start 2024-05-04T17:45:25.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f6739d84640 10 monclient: init 2024-05-04T17:45:25.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f15bb7fe640 10 monclient(hunting): handle_auth_done global_id 4327 payload 274 2024-05-04T17:45:25.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.399+0000 7f21cc570640 1 -- start start 2024-05-04T17:45:25.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.399+0000 7f21cc570640 10 monclient: init 2024-05-04T17:45:25.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f21cc570640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f21cc570640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-05-04T17:45:25.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f21cc570640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21c40a9720 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:25.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f21cc570640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21c40a9b00 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:25.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f21cc570640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f21c40aa890 0x7f21c40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f21cc570640 10 monclient(hunting): picked mon.noname-a con 0x7f21c40aa890 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:25.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f21cc570640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21c40abc40 con 0x7f21c40a9720 2024-05-04T17:45:25.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f21cc570640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21c4146fa0 con 0x7f21c40a9b00 2024-05-04T17:45:25.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f21cc570640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f21cc570640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f21c4146090 con 0x7f21c40aa890 2024-05-04T17:45:25.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f21cc570640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f21cc570640 10 monclient(hunting): authenticate will time out at 1347.514426s 2024-05-04T17:45:25.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f15bb7fe640 1 -- 172.21.15.19:0/1625872914 >> v1:172.21.15.186:6789/0 conn(0x7f15bc144180 legacy=0x7f15bc1445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f15bb7fe640 1 -- 172.21.15.19:0/1625872914 >> v1:172.21.15.19:6789/0 conn(0x7f15bc06f540 legacy=0x7f15bc06f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f15bb7fe640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f15bb7fe640 1 monclient: found mon.noname-c 2024-05-04T17:45:25.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.401+0000 7f6739d84640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f15bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.401+0000 7f6739d84640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-05-04T17:45:25.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.401+0000 7f6739d84640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f673406cd10 0x7f673413c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f15bb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.400+0000 7f15bb7fe640 1 -- 172.21.15.19:0/1625872914 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15bc073390 con 0x7f15bc072390 2024-05-04T17:45:25.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.401+0000 7f6739d84640 10 monclient(hunting): picked mon.noname-a con 0x7f673406cd10 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:25.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.401+0000 7f6739d84640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6734071b80 0x7f673413ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.401+0000 7f6739d84640 10 monclient(hunting): picked mon.noname-c con 0x7f6734071b80 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:25.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.401+0000 7f6739d84640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6734142d00 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:25.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.401+0000 7f6739d84640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67341438c0 con 0x7f6734142d00 2024-05-04T17:45:25.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f6739d84640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f6739d84640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f673413d3f0 con 0x7f673406cd10 2024-05-04T17:45:25.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f6739d84640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f6739d84640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6734072900 con 0x7f6734071b80 2024-05-04T17:45:25.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f02c2d9d640 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2024-05-04T17:45:25.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f02c2d9d640 1 -- 172.21.15.19:0/158049333 >> v2:172.21.15.186:3300/0 conn(0x7f02c40b2e90 msgr2=0x7f02c413fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f02c2d9d640 1 --2- 172.21.15.19:0/158049333 >> v2:172.21.15.186:3300/0 conn(0x7f02c40b2e90 0x7f02c413fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f6739d84640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f6739d84640 10 monclient(hunting): authenticate will time out at 1347.515916s 2024-05-04T17:45:25.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f02c2d9d640 1 -- 172.21.15.19:0/158049333 >> v1:172.21.15.186:6789/0 conn(0x7f02c40a9b00 legacy=0x7f02c4146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f02c2d9d640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f02c2d9d640 1 monclient: found mon.noname-b 2024-05-04T17:45:25.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f02c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f02c2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.402+0000 7f02c2d9d640 1 -- 172.21.15.19:0/158049333 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c4146370 con 0x7f02c40a9720 2024-05-04T17:45:25.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.404+0000 7f15bb7fe640 1 --2- 172.21.15.19:0/1625872914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15bc072390 0x7f15bc072770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009e50 tx=0x7f15ac02bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60c5b8462674c1be server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.404+0000 7f82fffff640 1 --2- 172.21.15.19:0/2551033273 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f83000a9b00 0x7f83000aa890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f82f8009e50 tx=0x7f82f802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f739f16e1d5759f server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.405+0000 7f02c2d9d640 1 --2- 172.21.15.19:0/158049333 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f02c40a9720 0x7f02c40aa890 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f02b40097a0 tx=0x7f02b402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbd7c25eb4d79a55 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.409+0000 7f7a2f7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7a30072390 0x7f7a30072770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.410+0000 7f93eb59e640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.410+0000 7f93eb59e640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.410+0000 7f93eb59e640 10 monclient: monmap: 2024-05-04T17:45:25.425 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.425 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.425 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.411405+0000 2024-05-04T17:45:25.425 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.411405+0000 2024-05-04T17:45:25.425 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.425 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.425 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.425 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.425 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.425 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.425 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.425 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f7a2ffff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7a3006f540 0x7f7a3006f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f93eb59e640 1 Processor -- start 2024-05-04T17:45:25.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f93eb59e640 1 -- start start 2024-05-04T17:45:25.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f93eb59e640 10 monclient: init 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f82ff7fe640 1 -- 172.21.15.19:0/2551033273 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f802ca70 con 0x7f83000a9b00 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f82ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f82ff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f82ff7fe640 10 monclient: dump: 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.427 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.428 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f82ff7fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f82ff7fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f82ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f8306dc7640 5 monclient: authenticate success, global_id 4256 2024-05-04T17:45:25.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.411+0000 7f8306dc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:25.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-05-04T17:45:25.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93e4036e30 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:25.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f93e40c3340 0x7f93e40c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f93e40c3340 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:25.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40355f0 0x7f93e40c9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f93e40355f0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93e402e6f0 con 0x7f93e4036e30 2024-05-04T17:45:25.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f93e4037210 con 0x7f93e40c3340 2024-05-04T17:45:25.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f93e402e9d0 con 0x7f93e40355f0 2024-05-04T17:45:25.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.412+0000 7f93eb59e640 10 monclient(hunting): authenticate will time out at 1347.526205s 2024-05-04T17:45:25.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 1 -- 172.21.15.19:0/158049333 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b4035020 con 0x7f02c40a9720 2024-05-04T17:45:25.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:25.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 10 monclient: dump: 2024-05-04T17:45:25.431 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.431 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.431 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c9175640 5 monclient: authenticate success, global_id 4259 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 1 -- 172.21.15.19:0/158049333 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02b4031810 con 0x7f02c40a9720 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c9175640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c9175640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c9175640 4 set_mon_vals no callback set 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 1 -- 172.21.15.19:0/158049333 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b4031be0 con 0x7f02c40a9720 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c1d9b640 10 monclient: dump: 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.433 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c9175640 10 monclient: shutdown 2024-05-04T17:45:25.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c9175640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c9175640 1 -- 172.21.15.19:0/158049333 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f02c40a9720 msgr2=0x7f02c40aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.413+0000 7f02c9175640 1 --2- 172.21.15.19:0/158049333 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f02c40a9720 0x7f02c40aa890 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f02b40097a0 tx=0x7f02b402b8b0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f02c9175640 1 -- 172.21.15.19:0/158049333 shutdown_connections 2024-05-04T17:45:25.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f02c9175640 1 --2- 172.21.15.19:0/158049333 >> v2:172.21.15.186:3300/0 conn(0x7f02c40b2e90 0x7f02c413fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f02c9175640 1 --2- 172.21.15.19:0/158049333 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f02c40a9720 0x7f02c40aa890 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f02c9175640 1 -- 172.21.15.19:0/158049333 >> 172.21.15.19:0/158049333 conn(0x7f02c4067780 msgr2=0x7f02c40b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f02c9175640 1 -- 172.21.15.19:0/158049333 shutdown_connections 2024-05-04T17:45:25.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f02c9175640 1 -- 172.21.15.19:0/158049333 wait complete. 2024-05-04T17:45:25.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f82ff7fe640 1 -- 172.21.15.19:0/2551033273 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f802cc10 con 0x7f83000a9b00 2024-05-04T17:45:25.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f82ff7fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f82ff7fe640 1 -- 172.21.15.19:0/2551033273 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f80343f0 con 0x7f83000a9b00 2024-05-04T17:45:25.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f82ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f82ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:25.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f82ff7fe640 10 monclient: dump: 2024-05-04T17:45:25.435 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.435 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.435 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f8306dc7640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f8306dc7640 4 set_mon_vals no callback set 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f8306dc7640 10 monclient: shutdown 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f8306dc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f8306dc7640 1 -- 172.21.15.19:0/2551033273 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f83000a9b00 msgr2=0x7f83000aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f8306dc7640 1 --2- 172.21.15.19:0/2551033273 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f83000a9b00 0x7f83000aa890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f82f8009e50 tx=0x7f82f802bed0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f02c9175640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.414+0000 7f02c9175640 10 monclient: monmap: 2024-05-04T17:45:25.437 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.437 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:25.437 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:25.437 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:25.437 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.437 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.437 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:25.437 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:25.437 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:25.437 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f8306dc7640 1 -- 172.21.15.19:0/2551033273 shutdown_connections 2024-05-04T17:45:25.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f8306dc7640 1 --2- 172.21.15.19:0/2551033273 >> v2:172.21.15.19:3300/0 conn(0x7f83000b2e90 0x7f830013fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f8306dc7640 1 --2- 172.21.15.19:0/2551033273 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f83000a9b00 0x7f83000aa890 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f8306dc7640 1 -- 172.21.15.19:0/2551033273 >> 172.21.15.19:0/2551033273 conn(0x7f8300067780 msgr2=0x7f83000b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f8306dc7640 1 -- 172.21.15.19:0/2551033273 shutdown_connections 2024-05-04T17:45:25.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f7a2ffff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7a3006f540 0x7f7a3006f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:36942/0 (socket says 172.21.15.19:36942) 2024-05-04T17:45:25.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f7a2ffff640 1 -- 172.21.15.19:0/402557651 learned_addr learned my addr 172.21.15.19:0/402557651 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f7a2ffff640 10 monclient(hunting): get_auth_request con 0x7f7a3006f540 auth_method 0 2024-05-04T17:45:25.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f7a2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f7a2ffff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f7a2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f02c9175640 1 librados: starting msgr at 2024-05-04T17:45:25.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f02c9175640 1 librados: starting objecter 2024-05-04T17:45:25.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f02c9175640 1 Processor -- start 2024-05-04T17:45:25.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f02c9175640 1 -- start start 2024-05-04T17:45:25.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f02c9175640 1 librados: setting wanted keys 2024-05-04T17:45:25.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f02c9175640 1 librados: calling monclient init 2024-05-04T17:45:25.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f02c9175640 10 monclient: init 2024-05-04T17:45:25.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.415+0000 7f8306dc7640 1 -- 172.21.15.19:0/2551033273 wait complete. 2024-05-04T17:45:25.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f8306dc7640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f8306dc7640 10 monclient: monmap: 2024-05-04T17:45:25.440 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.441 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:25.441 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:25.441 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:25.441 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.441 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.441 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:25.441 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:25.441 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:25.441 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:45:25.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02c40a9720 0x7f02c406ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 10 monclient(hunting): picked mon.noname-a con 0x7f02c40a9720 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f02c40a9b00 0x7f02c41d7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 10 monclient(hunting): picked mon.noname-c con 0x7f02c40a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02c40b2e90 0x7f02c41d77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 10 monclient(hunting): picked mon.noname-b con 0x7f02c40b2e90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f02c4140770 con 0x7f02c40a9720 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f02c41405f0 con 0x7f02c40b2e90 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f02c406e540 con 0x7f02c40a9b00 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f02c9175640 10 monclient(hunting): authenticate will time out at 1347.529991s 2024-05-04T17:45:25.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f8306dc7640 1 librados: starting msgr at 2024-05-04T17:45:25.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f8306dc7640 1 librados: starting objecter 2024-05-04T17:45:25.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f7a2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f7a2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.444 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f7a2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f8306dc7640 1 Processor -- start 2024-05-04T17:45:25.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f8306dc7640 1 -- start start 2024-05-04T17:45:25.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f8306dc7640 1 librados: setting wanted keys 2024-05-04T17:45:25.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f8306dc7640 1 librados: calling monclient init 2024-05-04T17:45:25.445 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.416+0000 7f8306dc7640 10 monclient: init 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83000a9720 0x7f83001db610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 10 monclient(hunting): picked mon.noname-b con 0x7f83000a9720 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83000a9b00 0x7f8300147fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 10 monclient(hunting): picked mon.noname-a con 0x7f83000a9b00 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f83000b2e90 0x7f83001484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 10 monclient(hunting): picked mon.noname-c con 0x7f83000b2e90 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8300140770 con 0x7f83000a9b00 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83001405f0 con 0x7f83000a9720 2024-05-04T17:45:25.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f830014d420 con 0x7f83000b2e90 2024-05-04T17:45:25.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.417+0000 7f8306dc7640 10 monclient(hunting): authenticate will time out at 1347.531544s 2024-05-04T17:45:25.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.419+0000 7f7a2ffff640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-05-04T17:45:25.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.419+0000 7f7a2ffff640 1 -- 172.21.15.19:0/402557651 >> v2:172.21.15.186:3300/0 conn(0x7f7a30072390 msgr2=0x7f7a30072770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.419+0000 7f7a2ffff640 1 --2- 172.21.15.19:0/402557651 >> v2:172.21.15.186:3300/0 conn(0x7f7a30072390 0x7f7a30072770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.419+0000 7f7a2ffff640 1 -- 172.21.15.19:0/402557651 >> v2:172.21.15.19:3300/0 conn(0x7f7a30144180 msgr2=0x7f7a301445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.419+0000 7f7a2ffff640 1 --2- 172.21.15.19:0/402557651 >> v2:172.21.15.19:3300/0 conn(0x7f7a30144180 0x7f7a301445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.419+0000 7f7a2ffff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.419+0000 7f7a2ffff640 1 monclient: found mon.noname-b 2024-05-04T17:45:25.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.419+0000 7f7a2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.419+0000 7f7a2ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.419+0000 7f7a2ffff640 1 -- 172.21.15.19:0/402557651 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a30073390 con 0x7f7a3006f540 2024-05-04T17:45:25.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.421+0000 7f02c259c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f02c40a9b00 0x7f02c41d7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.421+0000 7f93ead9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40355f0 0x7f93e40c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.423+0000 7f7a2ffff640 1 --2- 172.21.15.19:0/402557651 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7a3006f540 0x7f7a3006f9f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7a300aea60 tx=0x7f7a1802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0f09922e3ec839 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.423+0000 7f02c259c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f02c40a9b00 0x7f02c41d7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:36960/0 (socket says 172.21.15.19:36960) 2024-05-04T17:45:25.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.423+0000 7f02c259c640 1 -- 172.21.15.19:0/3067830237 learned_addr learned my addr 172.21.15.19:0/3067830237 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.423+0000 7f02c259c640 10 monclient(hunting): get_auth_request con 0x7f02c40a9b00 auth_method 0 2024-05-04T17:45:25.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.423+0000 7f02c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.423+0000 7f02c259c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.423+0000 7f02c259c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.424+0000 7f93ead9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40355f0 0x7f93e40c9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:36944/0 (socket says 172.21.15.19:36944) 2024-05-04T17:45:25.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.424+0000 7f93ead9d640 1 -- 172.21.15.19:0/3011613841 learned_addr learned my addr 172.21.15.19:0/3011613841 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.424+0000 7f93ead9d640 10 monclient(hunting): get_auth_request con 0x7f93e40355f0 auth_method 0 2024-05-04T17:45:25.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.424+0000 7f93ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.424+0000 7f93ead9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.424+0000 7f93ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.425+0000 7f02c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.425+0000 7f02c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.425+0000 7f02c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.451 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.428+0000 7f93ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.428+0000 7f93ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.428+0000 7f93ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.431+0000 7f93ead9d640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-05-04T17:45:25.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.431+0000 7f93ead9d640 1 -- 172.21.15.19:0/3011613841 >> v2:172.21.15.186:3300/0 conn(0x7f93e40c3340 msgr2=0x7f93e40c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.431+0000 7f93ead9d640 1 --2- 172.21.15.19:0/3011613841 >> v2:172.21.15.186:3300/0 conn(0x7f93e40c3340 0x7f93e40c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.431+0000 7f93ead9d640 1 -- 172.21.15.19:0/3011613841 >> v1:172.21.15.186:6789/0 conn(0x7f93e4036e30 legacy=0x7f93e40c2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.431+0000 7f93ead9d640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.431+0000 7f93ead9d640 1 monclient: found mon.noname-b 2024-05-04T17:45:25.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.431+0000 7f93ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.431+0000 7f93ead9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.431+0000 7f93ead9d640 1 -- 172.21.15.19:0/3011613841 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e402cff0 con 0x7f93e40355f0 2024-05-04T17:45:25.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.434+0000 7f02c259c640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2024-05-04T17:45:25.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.434+0000 7f02c259c640 1 -- 172.21.15.19:0/3067830237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02c40b2e90 msgr2=0x7f02c41d77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.434+0000 7f02c259c640 1 --2- 172.21.15.19:0/3067830237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02c40b2e90 0x7f02c41d77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.434+0000 7f02c259c640 1 -- 172.21.15.19:0/3067830237 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02c40a9720 msgr2=0x7f02c406ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.434+0000 7f02c259c640 1 --2- 172.21.15.19:0/3067830237 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f02c40a9720 0x7f02c406ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.434+0000 7f02c259c640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.434+0000 7f02c259c640 1 monclient: found mon.noname-c 2024-05-04T17:45:25.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.434+0000 7f02c259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.434+0000 7f02c259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.434+0000 7f02c259c640 1 -- 172.21.15.19:0/3067830237 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c4071100 con 0x7f02c40a9b00 2024-05-04T17:45:25.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 1 -- 172.21.15.19:0/402557651 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18035020 con 0x7f7a3006f540 2024-05-04T17:45:25.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:25.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 10 monclient: dump: 2024-05-04T17:45:25.455 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.455 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.455 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.455 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.455 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.455 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.455 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.455 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 1 -- 172.21.15.19:0/402557651 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a18031640 con 0x7f7a3006f540 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 1 -- 172.21.15.19:0/402557651 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1802d040 con 0x7f7a3006f540 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a2e7fc640 10 monclient: dump: 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.456 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a35af2640 5 monclient: authenticate success, global_id 4262 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a35af2640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.440+0000 7f7a35af2640 4 set_mon_vals no callback set 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 10 monclient: shutdown 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 1 -- 172.21.15.19:0/402557651 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7a3006f540 msgr2=0x7f7a3006f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 1 --2- 172.21.15.19:0/402557651 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7a3006f540 0x7f7a3006f9f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7a300aea60 tx=0x7f7a1802b8f0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 1 -- 172.21.15.19:0/402557651 shutdown_connections 2024-05-04T17:45:25.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 1 --2- 172.21.15.19:0/402557651 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7a3006f540 0x7f7a3006f9f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 1 --2- 172.21.15.19:0/402557651 >> v2:172.21.15.19:3300/0 conn(0x7f7a30144180 0x7f7a301445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 1 --2- 172.21.15.19:0/402557651 >> v2:172.21.15.186:3300/0 conn(0x7f7a30072390 0x7f7a30072770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 1 -- 172.21.15.19:0/402557651 >> 172.21.15.19:0/402557651 conn(0x7f7a300b4580 msgr2=0x7f7a300b4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 1 -- 172.21.15.19:0/402557651 shutdown_connections 2024-05-04T17:45:25.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 1 -- 172.21.15.19:0/402557651 wait complete. 2024-05-04T17:45:25.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f93ead9d640 1 --2- 172.21.15.19:0/3011613841 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40355f0 0x7f93e40c9480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f93ec02d750 tx=0x7f93ec0a6ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c5456a798bacc57 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.459 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.441+0000 7f7a35af2640 10 monclient: monmap: 2024-05-04T17:45:25.459 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.459 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:25.459 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:25.459 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:25.459 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.459 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.442+0000 7f7a35af2640 1 librados: starting msgr at 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.442+0000 7f7a35af2640 1 librados: starting objecter 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.442+0000 7f7a35af2640 1 Processor -- start 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.442+0000 7f7a35af2640 1 -- start start 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.442+0000 7f7a35af2640 1 librados: setting wanted keys 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.442+0000 7f7a35af2640 1 librados: calling monclient init 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.442+0000 7f7a35af2640 10 monclient: init 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7a3006f540 0x7f7a30130e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.460 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 10 monclient(hunting): picked mon.noname-c con 0x7f7a3006f540 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a30072390 0x7f7a300b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 10 monclient(hunting): picked mon.noname-b con 0x7f7a30072390 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7a30144180 0x7f7a3012d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 10 monclient(hunting): picked mon.noname-a con 0x7f7a30144180 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7a300700b0 con 0x7f7a30144180 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3006ff30 con 0x7f7a30072390 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f7a30070230 con 0x7f7a3006f540 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.443+0000 7f7a35af2640 10 monclient(hunting): authenticate will time out at 1347.556882s 2024-05-04T17:45:25.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.445+0000 7f02c259c640 1 --2- 172.21.15.19:0/3067830237 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f02c40a9b00 0x7f02c41d7270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00c8a0 tx=0x7f02bc00cd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.445+0000 7f02ab7fe640 10 client.?.objecter ms_handle_connect 0x7f02c40a9b00 2024-05-04T17:45:25.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.445+0000 7f02ab7fe640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.467+0000 7f21c9b0b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f21c40a9b00 0x7f21c413fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:46464/0 (socket says 172.21.15.19:46464) 2024-05-04T17:45:25.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.467+0000 7f21c9b0b640 1 -- 172.21.15.19:0/3649043952 learned_addr learned my addr 172.21.15.19:0/3649043952 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:25.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.467+0000 7f6732ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6734071b80 0x7f673413ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.468+0000 7f6732ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6734071b80 0x7f673413ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36258/0 (socket says 172.21.15.19:36258) 2024-05-04T17:45:25.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.468+0000 7f6732ffd640 1 -- 172.21.15.19:0/706864895 learned_addr learned my addr 172.21.15.19:0/706864895 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.468+0000 7f6732ffd640 10 monclient(hunting): get_auth_request con 0x7f6734071b80 auth_method 0 2024-05-04T17:45:25.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.468+0000 7f6732ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.468+0000 7f6732ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.468+0000 7f6732ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.469+0000 7f6732ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.469+0000 7f6732ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.469+0000 7f6732ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.471+0000 7f6732ffd640 10 monclient(hunting): handle_auth_done global_id 4333 payload 274 2024-05-04T17:45:25.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.471+0000 7f6732ffd640 1 -- 172.21.15.19:0/706864895 >> v2:172.21.15.19:3300/0 conn(0x7f673406cd10 msgr2=0x7f673413c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.471+0000 7f6732ffd640 1 --2- 172.21.15.19:0/706864895 >> v2:172.21.15.19:3300/0 conn(0x7f673406cd10 0x7f673413c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.471+0000 7f6732ffd640 1 -- 172.21.15.19:0/706864895 >> v1:172.21.15.186:6789/0 conn(0x7f6734142d00 legacy=0x7f67341431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.471+0000 7f6732ffd640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.471+0000 7f6732ffd640 1 monclient: found mon.noname-c 2024-05-04T17:45:25.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.471+0000 7f6732ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.471+0000 7f6732ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.471+0000 7f6732ffd640 1 -- 172.21.15.19:0/706864895 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6734072be0 con 0x7f6734071b80 2024-05-04T17:45:25.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.473+0000 7f82fffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83000a9b00 0x7f8300147fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.473+0000 7f21cab0d640 1 --2- 172.21.15.19:0/3649043952 >> v2:172.21.15.19:3300/0 conn(0x7f21c40aa890 0x7f21c40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.474+0000 7f21cab0d640 10 monclient(hunting): get_auth_request con 0x7f21c40aa890 auth_method 0 2024-05-04T17:45:25.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.474+0000 7f21cab0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.474+0000 7f21cab0d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.474+0000 7f21cab0d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f21cab0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f21cab0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f21cab0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f7769d9b640 1 --2- 172.21.15.19:0/4282957374 >> v2:172.21.15.19:3300/0 conn(0x7f776c153790 0x7f776c072010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15baffd640 1 -- 172.21.15.19:0/1625872914 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac02cb10 con 0x7f15bc072390 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15baffd640 10 monclient: dump: 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15baffd640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15baffd640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15c1e3e640 5 monclient: authenticate success, global_id 4327 2024-05-04T17:45:25.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15c1e3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:25.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15baffd640 1 -- 172.21.15.19:0/1625872914 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15ac02ccb0 con 0x7f15bc072390 2024-05-04T17:45:25.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15baffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15c1e3e640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:25.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.476+0000 7f15c1e3e640 4 set_mon_vals no callback set 2024-05-04T17:45:25.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f7769d9b640 10 monclient(hunting): get_auth_request con 0x7f776c153790 auth_method 0 2024-05-04T17:45:25.479 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f7769d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f7769d9b640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f7769d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f6732ffd640 1 --2- 172.21.15.19:0/706864895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6734071b80 0x7f673413ceb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6728009870 tx=0x7f672802b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=172d4f03e2883c95 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f15baffd640 1 -- 172.21.15.19:0/1625872914 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac02ce60 con 0x7f15bc072390 2024-05-04T17:45:25.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f15baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f15baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:25.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f15baffd640 10 monclient: dump: 2024-05-04T17:45:25.481 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.481 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.481 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.481 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.481 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f15c1e3e640 10 monclient: shutdown 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f15c1e3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f15c1e3e640 1 -- 172.21.15.19:0/1625872914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15bc072390 msgr2=0x7f15bc072770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f15c1e3e640 1 --2- 172.21.15.19:0/1625872914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15bc072390 0x7f15bc072770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009e50 tx=0x7f15ac02bf70 comp rx=0 tx=0).stop 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f15c1e3e640 1 -- 172.21.15.19:0/1625872914 shutdown_connections 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f15c1e3e640 1 --2- 172.21.15.19:0/1625872914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15bc072390 0x7f15bc072770 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.477+0000 7f15c1e3e640 1 -- 172.21.15.19:0/1625872914 >> 172.21.15.19:0/1625872914 conn(0x7f15bc0b4580 msgr2=0x7f15bc0b4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.478+0000 7f15c1e3e640 1 -- 172.21.15.19:0/1625872914 shutdown_connections 2024-05-04T17:45:25.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.478+0000 7f15c1e3e640 1 -- 172.21.15.19:0/1625872914 wait complete. 2024-05-04T17:45:25.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.478+0000 7f21cab0d640 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2024-05-04T17:45:25.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.478+0000 7f21cab0d640 1 -- 172.21.15.19:0/3649043952 >> v1:172.21.15.186:6789/0 conn(0x7f21c40a9b00 legacy=0x7f21c413fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.478+0000 7f21cab0d640 1 -- 172.21.15.19:0/3649043952 >> v1:172.21.15.19:6789/0 conn(0x7f21c40a9720 legacy=0x7f21c4146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.478+0000 7f21cab0d640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.478+0000 7f21cab0d640 1 monclient: found mon.noname-a 2024-05-04T17:45:25.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.478+0000 7f21cab0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.478+0000 7f21cab0d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.478+0000 7f21cab0d640 1 -- 172.21.15.19:0/3649043952 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c4146370 con 0x7f21c40aa890 2024-05-04T17:45:25.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f15c1e3e640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.484 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f15c1e3e640 10 monclient: monmap: 2024-05-04T17:45:25.484 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.484 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:25.484 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:25.484 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:25.484 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.484 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.484 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:25.484 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:25.484 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:25.485 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f7a2effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a30072390 0x7f7a300b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f7769d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f7769d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.485 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f7769d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f15c1e3e640 1 librados: starting msgr at 2024-05-04T17:45:25.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f15c1e3e640 1 librados: starting objecter 2024-05-04T17:45:25.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f15c1e3e640 1 Processor -- start 2024-05-04T17:45:25.486 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f15c1e3e640 1 -- start start 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f15c1e3e640 1 librados: setting wanted keys 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f15c1e3e640 1 librados: calling monclient init 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.479+0000 7f15c1e3e640 10 monclient: init 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f15bc06f540 0x7f15bc12ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 10 monclient(hunting): picked mon.noname-a con 0x7f15bc06f540 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f15bc072390 0x7f15bc12e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 10 monclient(hunting): picked mon.noname-c con 0x7f15bc072390 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15bc144180 0x7f15bc12b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 10 monclient(hunting): picked mon.noname-b con 0x7f15bc144180 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc152a30 con 0x7f15bc06f540 2024-05-04T17:45:25.487 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc1528b0 con 0x7f15bc144180 2024-05-04T17:45:25.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f15bc152bb0 con 0x7f15bc072390 2024-05-04T17:45:25.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.480+0000 7f15c1e3e640 10 monclient(hunting): authenticate will time out at 1347.594473s 2024-05-04T17:45:25.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7a2effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a30072390 0x7f7a300b3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36292/0 (socket says 172.21.15.19:36292) 2024-05-04T17:45:25.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7a2effd640 1 -- 172.21.15.19:0/3930593910 learned_addr learned my addr 172.21.15.19:0/3930593910 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7a2effd640 10 monclient(hunting): get_auth_request con 0x7f7a30072390 auth_method 0 2024-05-04T17:45:25.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7a2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7a2effd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.488 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7a2effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7769d9b640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-05-04T17:45:25.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7769d9b640 1 -- 172.21.15.19:0/4282957374 >> v1:172.21.15.186:6789/0 conn(0x7f776c06bba0 legacy=0x7f776c153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7769d9b640 1 -- 172.21.15.19:0/4282957374 >> v1:172.21.15.19:6789/0 conn(0x7f776c072550 legacy=0x7f776c159f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7769d9b640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7769d9b640 1 monclient: found mon.noname-a 2024-05-04T17:45:25.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7769d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7769d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.489 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.481+0000 7f7769d9b640 1 -- 172.21.15.19:0/4282957374 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f776c071910 con 0x7f776c153790 2024-05-04T17:45:25.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.482+0000 7f8304b63640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83000a9720 0x7f83001db610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.482+0000 7f82fffff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83000a9b00 0x7f8300147fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37214/0 (socket says 172.21.15.19:37214) 2024-05-04T17:45:25.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.482+0000 7f82fffff640 1 -- 172.21.15.19:0/1806753867 learned_addr learned my addr 172.21.15.19:0/1806753867 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.482+0000 7f82fffff640 10 monclient(hunting): get_auth_request con 0x7f83000a9b00 auth_method 0 2024-05-04T17:45:25.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.482+0000 7f82fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.482+0000 7f82fffff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.490 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.482+0000 7f82fffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.486+0000 7f21cab0d640 1 --2- 172.21.15.19:0/3649043952 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21c40aa890 0x7f21c40b2e90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009dc0 tx=0x7f21bc02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec81cc5cc930b0e server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.486+0000 7f7769d9b640 1 --2- 172.21.15.19:0/4282957374 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f776c153790 0x7f776c072010 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f776c0b1b30 tx=0x7f776002dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=736a067d9df9d768 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.487+0000 7f7a2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.491 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.487+0000 7f7a2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.487+0000 7f7a2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.487+0000 7f8304b63640 10 monclient(hunting): get_auth_request con 0x7f83000a9720 auth_method 0 2024-05-04T17:45:25.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.487+0000 7f8304b63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.487+0000 7f8304b63640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.487+0000 7f8304b63640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.488+0000 7f82fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.488+0000 7f82fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.488+0000 7f82fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.492 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.488+0000 7f8304b63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.488+0000 7f8304b63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.488+0000 7f8304b63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.491+0000 7f7a2effd640 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2024-05-04T17:45:25.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.491+0000 7f7a2effd640 1 -- 172.21.15.19:0/3930593910 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7a3006f540 msgr2=0x7f7a30130e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.493 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.491+0000 7f7a2effd640 1 --2- 172.21.15.19:0/3930593910 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f7a3006f540 0x7f7a30130e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.491+0000 7f7a2effd640 1 -- 172.21.15.19:0/3930593910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7a30144180 msgr2=0x7f7a3012d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.491+0000 7f7a2effd640 1 --2- 172.21.15.19:0/3930593910 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f7a30144180 0x7f7a3012d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.491+0000 7f7a2effd640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.491+0000 7f7a2effd640 1 monclient: found mon.noname-b 2024-05-04T17:45:25.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.491+0000 7f7a2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.491+0000 7f7a2effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.491+0000 7f7a2effd640 1 -- 172.21.15.19:0/3930593910 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a300aa0d0 con 0x7f7a30072390 2024-05-04T17:45:25.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.493+0000 7f82fffff640 10 monclient(hunting): handle_auth_done global_id 4287 payload 824 2024-05-04T17:45:25.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.493+0000 7f82fffff640 1 -- 172.21.15.19:0/1806753867 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f83000b2e90 msgr2=0x7f83001484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.494 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.493+0000 7f82fffff640 1 --2- 172.21.15.19:0/1806753867 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f83000b2e90 0x7f83001484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.493+0000 7f82fffff640 1 -- 172.21.15.19:0/1806753867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83000a9720 msgr2=0x7f83001db610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.493+0000 7f82fffff640 1 --2- 172.21.15.19:0/1806753867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83000a9720 0x7f83001db610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.493+0000 7f82fffff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.493+0000 7f82fffff640 1 monclient: found mon.noname-a 2024-05-04T17:45:25.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.493+0000 7f82fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.493+0000 7f82fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.495 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.493+0000 7f82fffff640 1 -- 172.21.15.19:0/1806753867 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8300149aa0 con 0x7f83000a9b00 2024-05-04T17:45:25.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.499+0000 7f82fffff640 1 --2- 172.21.15.19:0/1806753867 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83000a9b00 0x7f8300147fb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f82f8036040 tx=0x7f82f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.499+0000 7f82fd7fa640 10 client.?.objecter ms_handle_connect 0x7f83000a9b00 2024-05-04T17:45:25.500 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.499+0000 7f82fd7fa640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:25.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 1 -- 172.21.15.19:0/706864895 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728038070 con 0x7f6734071b80 2024-05-04T17:45:25.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:25.504 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 10 monclient: dump: 2024-05-04T17:45:25.504 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.504 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.504 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.504 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.504 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 1 -- 172.21.15.19:0/706864895 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f672802f050 con 0x7f6734071b80 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 1 -- 172.21.15.19:0/706864895 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728031970 con 0x7f6734071b80 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f67327fc640 10 monclient: dump: 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.505 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.506 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.506 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.506 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.506 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.506 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.506 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.506 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.506 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f6739d84640 5 monclient: authenticate success, global_id 4333 2024-05-04T17:45:25.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f6739d84640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:25.506 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.503+0000 7f6739d84640 4 set_mon_vals no callback set 2024-05-04T17:45:25.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.504+0000 7f6739d84640 10 monclient: shutdown 2024-05-04T17:45:25.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.504+0000 7f6739d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.504+0000 7f6739d84640 1 -- 172.21.15.19:0/706864895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6734071b80 msgr2=0x7f673413ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.504+0000 7f6739d84640 1 --2- 172.21.15.19:0/706864895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6734071b80 0x7f673413ceb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6728009870 tx=0x7f672802b7f0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.504+0000 7f6739d84640 1 -- 172.21.15.19:0/706864895 shutdown_connections 2024-05-04T17:45:25.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.504+0000 7f6739d84640 1 --2- 172.21.15.19:0/706864895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6734071b80 0x7f673413ceb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.504+0000 7f6739d84640 1 --2- 172.21.15.19:0/706864895 >> v2:172.21.15.19:3300/0 conn(0x7f673406cd10 0x7f673413c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.504+0000 7f6739d84640 1 -- 172.21.15.19:0/706864895 >> 172.21.15.19:0/706864895 conn(0x7f67340b25e0 msgr2=0x7f673406c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.505+0000 7f6739d84640 1 -- 172.21.15.19:0/706864895 shutdown_connections 2024-05-04T17:45:25.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.505+0000 7f6739d84640 1 -- 172.21.15.19:0/706864895 wait complete. 2024-05-04T17:45:25.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.505+0000 7f15bbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15bc144180 0x7f15bc12b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.505+0000 7f6739d84640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f6739d84640 10 monclient: monmap: 2024-05-04T17:45:25.508 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.509 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:25.509 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:25.509 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:25.509 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.509 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.509 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:25.509 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:25.509 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:25.509 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f15bbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15bc144180 0x7f15bc12b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36304/0 (socket says 172.21.15.19:36304) 2024-05-04T17:45:25.509 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f15bbfff640 1 -- 172.21.15.19:0/3788873225 learned_addr learned my addr 172.21.15.19:0/3788873225 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f15bbfff640 10 monclient(hunting): get_auth_request con 0x7f15bc144180 auth_method 0 2024-05-04T17:45:25.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f15bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f15bbfff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f15bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.510 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f6739d84640 1 librados: starting msgr at 2024-05-04T17:45:25.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f6739d84640 1 librados: starting objecter 2024-05-04T17:45:25.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f6739d84640 1 Processor -- start 2024-05-04T17:45:25.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f6739d84640 1 -- start start 2024-05-04T17:45:25.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f6739d84640 1 librados: setting wanted keys 2024-05-04T17:45:25.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f6739d84640 1 librados: calling monclient init 2024-05-04T17:45:25.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.506+0000 7f6739d84640 10 monclient: init 2024-05-04T17:45:25.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.507+0000 7f6739d84640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.507+0000 7f7a2effd640 1 --2- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a30072390 0x7f7a300b3070 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7a2000b9c0 tx=0x7f7a2000be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.507+0000 7f7a27fff640 10 client.?.objecter ms_handle_connect 0x7f7a30072390 2024-05-04T17:45:25.511 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.507+0000 7f7a27fff640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:25.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.507+0000 7f6739d84640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-04T17:45:25.512 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f673406cd10 0x7f6734141960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 10 monclient(hunting): picked mon.noname-c con 0x7f673406cd10 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6734142d00 0x7f67341d6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 10 monclient(hunting): picked mon.noname-a con 0x7f6734142d00 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67341423e0 0x7f67341d70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 10 monclient(hunting): picked mon.noname-b con 0x7f67341423e0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f15bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f15bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.513 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f15bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f673414a090 con 0x7f6734142d00 2024-05-04T17:45:25.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6734073480 con 0x7f67341423e0 2024-05-04T17:45:25.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.514 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f673414a210 con 0x7f673406cd10 2024-05-04T17:45:25.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.508+0000 7f6739d84640 10 monclient(hunting): authenticate will time out at 1347.622126s 2024-05-04T17:45:25.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.511+0000 7f15bbfff640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2024-05-04T17:45:25.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.511+0000 7f15bbfff640 1 -- 172.21.15.19:0/3788873225 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f15bc072390 msgr2=0x7f15bc12e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.511+0000 7f15bbfff640 1 --2- 172.21.15.19:0/3788873225 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f15bc072390 0x7f15bc12e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.511+0000 7f15bbfff640 1 -- 172.21.15.19:0/3788873225 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f15bc06f540 msgr2=0x7f15bc12ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.511+0000 7f15bbfff640 1 --2- 172.21.15.19:0/3788873225 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f15bc06f540 0x7f15bc12ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.511+0000 7f15bbfff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.511+0000 7f15bbfff640 1 monclient: found mon.noname-b 2024-05-04T17:45:25.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.511+0000 7f15bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.512+0000 7f15bbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.512+0000 7f15bbfff640 1 -- 172.21.15.19:0/3788873225 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15bc12d1f0 con 0x7f15bc144180 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 1 -- 172.21.15.19:0/3649043952 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc039070 con 0x7f21c40aa890 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 10 monclient: dump: 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.517 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 1 -- 172.21.15.19:0/3649043952 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21bc02cc40 con 0x7f21c40aa890 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21cc570640 5 monclient: authenticate success, global_id 4281 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21cc570640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21cc570640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 1 -- 172.21.15.19:0/3649043952 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21bc034440 con 0x7f21c40aa890 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21cc570640 4 set_mon_vals no callback set 2024-05-04T17:45:25.518 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:25.519 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.515+0000 7f21c930a640 10 monclient: dump: 2024-05-04T17:45:25.519 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.519 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.519 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.519 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.519 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.519 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.519 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.519 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.519 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.520 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.516+0000 7f21cc570640 10 monclient: shutdown 2024-05-04T17:45:25.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.516+0000 7f21cc570640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.516+0000 7f21cc570640 1 -- 172.21.15.19:0/3649043952 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21c40aa890 msgr2=0x7f21c40b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.516+0000 7f21cc570640 1 --2- 172.21.15.19:0/3649043952 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21c40aa890 0x7f21c40b2e90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009dc0 tx=0x7f21bc02be90 comp rx=0 tx=0).stop 2024-05-04T17:45:25.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.516+0000 7f21cc570640 1 -- 172.21.15.19:0/3649043952 shutdown_connections 2024-05-04T17:45:25.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.516+0000 7f21cc570640 1 --2- 172.21.15.19:0/3649043952 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21c40aa890 0x7f21c40b2e90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.516+0000 7f21cc570640 1 -- 172.21.15.19:0/3649043952 >> 172.21.15.19:0/3649043952 conn(0x7f21c4067780 msgr2=0x7f21c40b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.516+0000 7f21cc570640 1 -- 172.21.15.19:0/3649043952 shutdown_connections 2024-05-04T17:45:25.520 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.516+0000 7f21cc570640 1 -- 172.21.15.19:0/3649043952 wait complete. 2024-05-04T17:45:25.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.517+0000 7f21cc570640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.521 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.517+0000 7f21cc570640 10 monclient: monmap: 2024-05-04T17:45:25.521 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.521 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:25.521 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:25.521 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.517+0000 7f21cc570640 1 librados: starting msgr at 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.517+0000 7f21cc570640 1 librados: starting objecter 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.517+0000 7f21cc570640 1 Processor -- start 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.517+0000 7f21cc570640 1 -- start start 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.517+0000 7f21cc570640 1 librados: setting wanted keys 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.517+0000 7f21cc570640 1 librados: calling monclient init 2024-05-04T17:45:25.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.517+0000 7f21cc570640 10 monclient: init 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21c40a9720 0x7f21c4140c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 10 monclient(hunting): picked mon.noname-b con 0x7f21c40a9720 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21c40a9b00 0x7f21c40b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 10 monclient(hunting): picked mon.noname-a con 0x7f21c40a9b00 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21c40aa890 0x7f21c41e0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 10 monclient(hunting): picked mon.noname-c con 0x7f21c40aa890 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f21c414e5b0 con 0x7f21c40a9b00 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f21c414e430 con 0x7f21c40a9720 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f21c414e730 con 0x7f21c40aa890 2024-05-04T17:45:25.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f21cc570640 10 monclient(hunting): authenticate will time out at 1347.631961s 2024-05-04T17:45:25.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f15bbfff640 1 --2- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15bc144180 0x7f15bc12b0e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f15b400c390 tx=0x7f15b400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f15b8ff9640 10 client.?.objecter ms_handle_connect 0x7f15bc144180 2024-05-04T17:45:25.524 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.518+0000 7f15b8ff9640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:25.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.519+0000 7f6733fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67341423e0 0x7f67341d70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.519+0000 7f6733fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67341423e0 0x7f67341d70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36306/0 (socket says 172.21.15.19:36306) 2024-05-04T17:45:25.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.519+0000 7f6733fff640 1 -- 172.21.15.19:0/908760950 learned_addr learned my addr 172.21.15.19:0/908760950 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.519+0000 7f6733fff640 10 monclient(hunting): get_auth_request con 0x7f67341423e0 auth_method 0 2024-05-04T17:45:25.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.519+0000 7f6733fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.519+0000 7f6733fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.525 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.519+0000 7f6733fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 1 -- 172.21.15.19:0/3930593910 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20019070 con 0x7f7a30072390 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 10 monclient: dump: 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.526 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 1 -- 172.21.15.19:0/3930593910 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a2000cba0 con 0x7f7a30072390 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 1 -- 172.21.15.19:0/3930593910 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20014440 con 0x7f7a30072390 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a27fff640 10 monclient: dump: 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.527 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.528 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.528 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.528 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f7a2dffb640 4 set_mon_vals no callback set 2024-05-04T17:45:25.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f21ca30c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21c40a9720 0x7f21c4140c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f776959a640 1 -- 172.21.15.19:0/4282957374 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776003b070 con 0x7f776c153790 2024-05-04T17:45:25.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f776959a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f776959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:25.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f776959a640 10 monclient: dump: 2024-05-04T17:45:25.529 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.529 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.529 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.529 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.529 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.529 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.529 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.529 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f776959a640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f776959a640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f776959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f776959a640 1 -- 172.21.15.19:0/4282957374 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f776002eb10 con 0x7f776c153790 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f776b59e640 5 monclient: authenticate success, global_id 4284 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f776b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f776959a640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.520+0000 7f776b59e640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776b59e640 4 set_mon_vals no callback set 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776959a640 1 -- 172.21.15.19:0/4282957374 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776000b040 con 0x7f776c153790 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776959a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:25.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776959a640 10 monclient: dump: 2024-05-04T17:45:25.531 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.531 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.531 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.531 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.531 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.531 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.531 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.531 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.531 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.531 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f7a35af2640 5 monclient: authenticate success, global_id 4339 2024-05-04T17:45:25.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f7a35af2640 10 monclient: _renew_subs 2024-05-04T17:45:25.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f7a35af2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f7a35af2640 1 -- 172.21.15.19:0/3930593910 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a300aa3b0 con 0x7f7a30072390 2024-05-04T17:45:25.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f7a35af2640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f7a35af2640 10 monclient: _renew_subs 2024-05-04T17:45:25.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f7a35af2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f7a35af2640 1 -- 172.21.15.19:0/3930593910 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a3012e9d0 con 0x7f7a30072390 2024-05-04T17:45:25.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f7a35af2640 1 librados: init done 2024-05-04T17:45:25.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776b59e640 10 monclient: shutdown 2024-05-04T17:45:25.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776b59e640 1 -- 172.21.15.19:0/4282957374 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f776c153790 msgr2=0x7f776c072010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776b59e640 1 --2- 172.21.15.19:0/4282957374 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f776c153790 0x7f776c072010 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f776c0b1b30 tx=0x7f776002dd60 comp rx=0 tx=0).stop 2024-05-04T17:45:25.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f6733fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f6733fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f6733fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776b59e640 1 -- 172.21.15.19:0/4282957374 shutdown_connections 2024-05-04T17:45:25.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776b59e640 1 --2- 172.21.15.19:0/4282957374 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f776c153790 0x7f776c072010 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776b59e640 1 -- 172.21.15.19:0/4282957374 >> 172.21.15.19:0/4282957374 conn(0x7f776c0b1120 msgr2=0x7f776c06b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776b59e640 1 -- 172.21.15.19:0/4282957374 shutdown_connections 2024-05-04T17:45:25.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.521+0000 7f776b59e640 1 -- 172.21.15.19:0/4282957374 wait complete. 2024-05-04T17:45:25.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f6732ffd640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6734142d00 0x7f67341d6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f7a35af2640 10 client.4339.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f7a35af2640 15 client.4339.objecter _session_command_op_assign -1 1 2024-05-04T17:45:25.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f7a35af2640 20 client.4339.objecter _get_session osd=-1 returning homeless 2024-05-04T17:45:25.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f7a35af2640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f7a35af2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f7a35af2640 1 -- 172.21.15.19:0/3930593910 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7a3012ecf0 con 0x7f7a30072390 2024-05-04T17:45:25.535 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f6732ffd640 10 monclient(hunting): get_auth_request con 0x7f6734142d00 auth_method 0 2024-05-04T17:45:25.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f776b59e640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.536 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f776b59e640 10 monclient: monmap: 2024-05-04T17:45:25.536 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.536 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:25.536 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:25.536 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:25.536 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.536 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.536 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:25.536 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:25.536 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:25.536 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f6732ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f6732ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.522+0000 7f6732ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 1 librados: starting msgr at 2024-05-04T17:45:25.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 1 librados: starting objecter 2024-05-04T17:45:25.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 1 Processor -- start 2024-05-04T17:45:25.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f6732ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f6732ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.538 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f6732ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 1 -- start start 2024-05-04T17:45:25.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 1 librados: setting wanted keys 2024-05-04T17:45:25.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 1 librados: calling monclient init 2024-05-04T17:45:25.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 10 monclient: init 2024-05-04T17:45:25.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T17:45:25.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f776c06bba0 0x7f776c08a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f776c06bba0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f776c072550 0x7f776c14e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.539 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f776c072550 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.523+0000 7f776b59e640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f776c153790 0x7f776c14e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f776b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f776c153790 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f776b59e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f776b59e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f776c06f450 con 0x7f776c072550 2024-05-04T17:45:25.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f776b59e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f776b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f776c06f5d0 con 0x7f776c06bba0 2024-05-04T17:45:25.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f776b59e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f776b59e640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f776c072a00 con 0x7f776c153790 2024-05-04T17:45:25.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f776b59e640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f776b59e640 10 monclient(hunting): authenticate will time out at 1347.637766s 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f93e959a640 1 -- 172.21.15.19:0/3011613841 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ec0a7a10 con 0x7f93e40355f0 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f93e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f93e959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f93e959a640 10 monclient: dump: 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.541 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f93e959a640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f93e959a640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f93e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f93eb59e640 5 monclient: authenticate success, global_id 4268 2024-05-04T17:45:25.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.524+0000 7f93eb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:25.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f6733fff640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2024-05-04T17:45:25.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f6733fff640 1 -- 172.21.15.19:0/908760950 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f673406cd10 msgr2=0x7f6734141960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f6733fff640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f673406cd10 0x7f6734141960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f6733fff640 1 -- 172.21.15.19:0/908760950 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6734142d00 msgr2=0x7f67341d6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f6733fff640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6734142d00 0x7f67341d6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f6733fff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f6733fff640 1 monclient: found mon.noname-b 2024-05-04T17:45:25.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f6733fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f6733fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f6733fff640 1 -- 172.21.15.19:0/908760950 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6734140ad0 con 0x7f67341423e0 2024-05-04T17:45:25.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f21ca30c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21c40a9720 0x7f21c4140c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36312/0 (socket says 172.21.15.19:36312) 2024-05-04T17:45:25.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f21ca30c640 1 -- 172.21.15.19:0/1644027812 learned_addr learned my addr 172.21.15.19:0/1644027812 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f21ca30c640 10 monclient(hunting): get_auth_request con 0x7f21c40a9720 auth_method 0 2024-05-04T17:45:25.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f21ca30c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f21ca30c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.526+0000 7f21ca30c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.528+0000 7f93e959a640 1 -- 172.21.15.19:0/3011613841 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93ec0a7c30 con 0x7f93e40355f0 2024-05-04T17:45:25.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.528+0000 7f93e959a640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.528+0000 7f93e959a640 1 -- 172.21.15.19:0/3011613841 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ec0af640 con 0x7f93e40355f0 2024-05-04T17:45:25.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.528+0000 7f93eb59e640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:25.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.528+0000 7f93eb59e640 4 set_mon_vals no callback set 2024-05-04T17:45:25.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.528+0000 7f93e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.528+0000 7f93e959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:25.545 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.528+0000 7f93e959a640 10 monclient: dump: 2024-05-04T17:45:25.545 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.545 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.546 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.546 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.546 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.546 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.546 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.546 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.546 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.546 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f93eb59e640 10 monclient: shutdown 2024-05-04T17:45:25.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f93eb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f93eb59e640 1 -- 172.21.15.19:0/3011613841 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40355f0 msgr2=0x7f93e40c9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.546 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f93eb59e640 1 --2- 172.21.15.19:0/3011613841 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40355f0 0x7f93e40c9480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f93ec02d750 tx=0x7f93ec0a6ed0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 1 -- 172.21.15.19:0/1806753867 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f8043020 con 0x7f83000a9b00 2024-05-04T17:45:25.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 10 monclient: dump: 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 1 -- 172.21.15.19:0/1806753867 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f803e450 con 0x7f83000a9b00 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 1 -- 172.21.15.19:0/1806753867 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f803e840 con 0x7f83000a9b00 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82fd7fa640 10 monclient: dump: 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f82feffd640 4 set_mon_vals no callback set 2024-05-04T17:45:25.549 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f8306dc7640 5 monclient: authenticate success, global_id 4287 2024-05-04T17:45:25.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f8306dc7640 10 monclient: _renew_subs 2024-05-04T17:45:25.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f8306dc7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f8306dc7640 1 -- 172.21.15.19:0/1806753867 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8300149d80 con 0x7f83000a9b00 2024-05-04T17:45:25.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f8306dc7640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f8306dc7640 10 monclient: _renew_subs 2024-05-04T17:45:25.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f8306dc7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f8306dc7640 1 -- 172.21.15.19:0/1806753867 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8300148a30 con 0x7f83000a9b00 2024-05-04T17:45:25.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f8306dc7640 1 librados: init done 2024-05-04T17:45:25.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f21ca30c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.529+0000 7f21ca30c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.550 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f21ca30c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f6733fff640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67341423e0 0x7f67341d70c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f672c00b990 tx=0x7f672c00be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f671bfff640 10 client.?.objecter ms_handle_connect 0x7f67341423e0 2024-05-04T17:45:25.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f671bfff640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:25.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f93eb59e640 1 -- 172.21.15.19:0/3011613841 shutdown_connections 2024-05-04T17:45:25.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f93eb59e640 1 --2- 172.21.15.19:0/3011613841 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40355f0 0x7f93e40c9480 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f93eb59e640 1 --2- 172.21.15.19:0/3011613841 >> v2:172.21.15.186:3300/0 conn(0x7f93e40c3340 0x7f93e40c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f93eb59e640 1 -- 172.21.15.19:0/3011613841 >> 172.21.15.19:0/3011613841 conn(0x7f93e40063d0 msgr2=0x7f93e40067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f93eb59e640 1 -- 172.21.15.19:0/3011613841 shutdown_connections 2024-05-04T17:45:25.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15b8ff9640 1 -- 172.21.15.19:0/3788873225 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b4017070 con 0x7f15bc144180 2024-05-04T17:45:25.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15b8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:25.552 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15b8ff9640 10 monclient: dump: 2024-05-04T17:45:25.552 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.552 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.552 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.552 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.552 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15b8ff9640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15b8ff9640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15c1e3e640 5 monclient: authenticate success, global_id 4345 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15c1e3e640 10 monclient: _renew_subs 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15c1e3e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15c1e3e640 1 -- 172.21.15.19:0/3788873225 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15bc12c4f0 con 0x7f15bc144180 2024-05-04T17:45:25.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f93eb59e640 1 -- 172.21.15.19:0/3011613841 wait complete. 2024-05-04T17:45:25.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15c1e3e640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15c1e3e640 10 monclient: _renew_subs 2024-05-04T17:45:25.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15c1e3e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15c1e3e640 1 -- 172.21.15.19:0/3788873225 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15bc12ed80 con 0x7f15bc144180 2024-05-04T17:45:25.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.530+0000 7f15c1e3e640 1 librados: init done 2024-05-04T17:45:25.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f15b8ff9640 1 -- 172.21.15.19:0/3788873225 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15b40035f0 con 0x7f15bc144180 2024-05-04T17:45:25.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f15b8ff9640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f8306dc7640 10 client.4287.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f8306dc7640 15 client.4287.objecter _session_command_op_assign -1 1 2024-05-04T17:45:25.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f8306dc7640 20 client.4287.objecter _get_session osd=-1 returning homeless 2024-05-04T17:45:25.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f8306dc7640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f8306dc7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f8306dc7640 1 -- 172.21.15.19:0/1806753867 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8300147c90 con 0x7f83000a9b00 2024-05-04T17:45:25.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f15ba7fc640 4 set_mon_vals no callback set 2024-05-04T17:45:25.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f15b8ff9640 1 -- 172.21.15.19:0/3788873225 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b40039e0 con 0x7f15bc144180 2024-05-04T17:45:25.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f15b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f15b8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:25.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f15b8ff9640 10 monclient: dump: 2024-05-04T17:45:25.556 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.556 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.556 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.556 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.556 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.556 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.556 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.556 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.556 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.556 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f93eb59e640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f93eb59e640 10 monclient: monmap: 2024-05-04T17:45:25.557 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.557 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:25.557 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:25.557 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:25.557 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.557 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.557 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:25.557 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:25.557 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:25.557 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f15c1e3e640 10 client.4345.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f15c1e3e640 15 client.4345.objecter _session_command_op_assign -1 1 2024-05-04T17:45:25.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f15c1e3e640 20 client.4345.objecter _get_session osd=-1 returning homeless 2024-05-04T17:45:25.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f15c1e3e640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.531+0000 7f15c1e3e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.532+0000 7f15c1e3e640 1 -- 172.21.15.19:0/3788873225 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f15a0002910 con 0x7f15bc144180 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.532+0000 7f93eb59e640 1 librados: starting msgr at 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.532+0000 7f93eb59e640 1 librados: starting objecter 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.532+0000 7f93eb59e640 1 Processor -- start 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.532+0000 7f93eb59e640 1 -- start start 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.532+0000 7f93eb59e640 1 librados: setting wanted keys 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.532+0000 7f93eb59e640 1 librados: calling monclient init 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.532+0000 7f93eb59e640 10 monclient: init 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f93e40355f0 0x7f93e40c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f93e40355f0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93e4036e30 0x7f93e40c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f93e4036e30 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.560 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40c3340 0x7f93e40cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f93e40c3340 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f93e40d02d0 con 0x7f93e40355f0 2024-05-04T17:45:25.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f93e40c9c50 con 0x7f93e4036e30 2024-05-04T17:45:25.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f93e40d0450 con 0x7f93e40c3340 2024-05-04T17:45:25.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.533+0000 7f93eb59e640 10 monclient(hunting): authenticate will time out at 1347.647284s 2024-05-04T17:45:25.561 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.534+0000 7f776ad9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f776c153790 0x7f776c14e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.535+0000 7f02ab7fe640 1 -- 172.21.15.19:0/3067830237 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc003b70 con 0x7f02c40a9b00 2024-05-04T17:45:25.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.535+0000 7f02ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.535+0000 7f02ab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:25.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.535+0000 7f02ab7fe640 10 monclient: dump: 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.535+0000 7f02ab7fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.535+0000 7f02ab7fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.535+0000 7f02ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.535+0000 7f02ab7fe640 1 -- 172.21.15.19:0/3067830237 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02bc003d30 con 0x7f02c40a9b00 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.535+0000 7f02ab7fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02ab7fe640 1 -- 172.21.15.19:0/3067830237 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc00aa90 con 0x7f02c40a9b00 2024-05-04T17:45:25.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02ab7fe640 10 monclient: dump: 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 5 monclient: authenticate success, global_id 4265 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 10 monclient: _renew_subs 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c159a640 4 set_mon_vals no callback set 2024-05-04T17:45:25.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 1 -- 172.21.15.19:0/3067830237 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c4071370 con 0x7f02c40a9b00 2024-05-04T17:45:25.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 10 monclient: _renew_subs 2024-05-04T17:45:25.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 1 -- 172.21.15.19:0/3067830237 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c406f4c0 con 0x7f02c40a9b00 2024-05-04T17:45:25.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 1 librados: init done 2024-05-04T17:45:25.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f776ad9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f776c153790 0x7f776c14e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:37004/0 (socket says 172.21.15.19:37004) 2024-05-04T17:45:25.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f776ad9d640 1 -- 172.21.15.19:0/3138761156 learned_addr learned my addr 172.21.15.19:0/3138761156 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f776ad9d640 10 monclient(hunting): get_auth_request con 0x7f776c153790 auth_method 0 2024-05-04T17:45:25.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f776ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f776ad9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f776ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f21cab0d640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21c40aa890 0x7f21c41e0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 10 client.4265.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 15 client.4265.objecter _session_command_op_assign -1 1 2024-05-04T17:45:25.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 20 client.4265.objecter _get_session osd=-1 returning homeless 2024-05-04T17:45:25.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.536+0000 7f02c9175640 1 -- 172.21.15.19:0/3067830237 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f02c4070100 con 0x7f02c40a9b00 2024-05-04T17:45:25.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.537+0000 7f21ca30c640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2024-05-04T17:45:25.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.537+0000 7f21ca30c640 1 -- 172.21.15.19:0/1644027812 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21c40aa890 msgr2=0x7f21c41e0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.537+0000 7f21ca30c640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f21c40aa890 0x7f21c41e0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.537+0000 7f21ca30c640 1 -- 172.21.15.19:0/1644027812 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21c40a9b00 msgr2=0x7f21c40b7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.537+0000 7f21ca30c640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f21c40a9b00 0x7f21c40b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.537+0000 7f21ca30c640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.537+0000 7f21ca30c640 1 monclient: found mon.noname-b 2024-05-04T17:45:25.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.537+0000 7f21ca30c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.537+0000 7f21ca30c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.537+0000 7f21ca30c640 1 -- 172.21.15.19:0/1644027812 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c413ceb0 con 0x7f21c40a9720 2024-05-04T17:45:25.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.538+0000 7f776ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.538+0000 7f776ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.538+0000 7f776ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.542+0000 7f776ad9d640 10 monclient(hunting): handle_auth_done global_id 4271 payload 824 2024-05-04T17:45:25.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.542+0000 7f776ad9d640 1 -- 172.21.15.19:0/3138761156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f776c06bba0 msgr2=0x7f776c08a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.542+0000 7f776ad9d640 1 --2- 172.21.15.19:0/3138761156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f776c06bba0 0x7f776c08a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.542+0000 7f776ad9d640 1 -- 172.21.15.19:0/3138761156 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f776c072550 msgr2=0x7f776c14e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.542+0000 7f776ad9d640 1 --2- 172.21.15.19:0/3138761156 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f776c072550 0x7f776c14e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.542+0000 7f776ad9d640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.542+0000 7f776ad9d640 1 monclient: found mon.noname-c 2024-05-04T17:45:25.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.542+0000 7f776ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.542+0000 7f776ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.542+0000 7f776ad9d640 1 -- 172.21.15.19:0/3138761156 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f776c14eb20 con 0x7f776c153790 2024-05-04T17:45:25.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.545+0000 7f776ad9d640 1 --2- 172.21.15.19:0/3138761156 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f776c153790 0x7f776c14e5e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f776400c7e0 tx=0x7f776400cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.545+0000 7f775affd640 10 client.?.objecter ms_handle_connect 0x7f776c153790 2024-05-04T17:45:25.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.545+0000 7f775affd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:25.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.547+0000 7f93ead9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40c3340 0x7f93e40cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.548+0000 7f21ca30c640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21c40a9720 0x7f21c4140c00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f21c000c470 tx=0x7f21c000c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.548+0000 7f21baffd640 10 client.?.objecter ms_handle_connect 0x7f21c40a9720 2024-05-04T17:45:25.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.548+0000 7f21baffd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:25.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.555+0000 7f93ead9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40c3340 0x7f93e40cb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:37018/0 (socket says 172.21.15.19:37018) 2024-05-04T17:45:25.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.555+0000 7f93ead9d640 1 -- 172.21.15.19:0/1360227486 learned_addr learned my addr 172.21.15.19:0/1360227486 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.555+0000 7f93ead9d640 10 monclient(hunting): get_auth_request con 0x7f93e40c3340 auth_method 0 2024-05-04T17:45:25.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.555+0000 7f93ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.555+0000 7f93ead9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.555+0000 7f93ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.556+0000 7f82fd7fa640 1 -- 172.21.15.19:0/1806753867 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f82f8046020 con 0x7f83000a9b00 2024-05-04T17:45:25.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.556+0000 7f82fd7fa640 1 --2- 172.21.15.19:0/1806753867 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f82ec06d810 0x7f82ec06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.556+0000 7f8304b63640 1 --2- 172.21.15.19:0/1806753867 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f82ec06d810 0x7f82ec06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.556+0000 7f8304b63640 10 monclient: get_auth_request con 0x7f82ec06d810 auth_method 0 2024-05-04T17:45:25.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.557+0000 7f8304b63640 1 --2- 172.21.15.19:0/1806753867 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f82ec06d810 0x7f82ec06fcd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f82f0006fd0 tx=0x7f82f0009060 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.557+0000 7f82fd7fa640 10 client.4287.objecter ms_handle_connect 0x7f82ec06d810 2024-05-04T17:45:25.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.558+0000 7f93ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.558+0000 7f93ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.558+0000 7f93ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f93ead9d640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-05-04T17:45:25.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f93ead9d640 1 -- 172.21.15.19:0/1360227486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93e4036e30 msgr2=0x7f93e40c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f93ead9d640 1 --2- 172.21.15.19:0/1360227486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93e4036e30 0x7f93e40c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f93ead9d640 1 -- 172.21.15.19:0/1360227486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f93e40355f0 msgr2=0x7f93e40c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f93ead9d640 1 --2- 172.21.15.19:0/1360227486 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f93e40355f0 0x7f93e40c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f93ead9d640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f93ead9d640 1 monclient: found mon.noname-c 2024-05-04T17:45:25.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f93ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f93ead9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f93ead9d640 1 -- 172.21.15.19:0/1360227486 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e40cd870 con 0x7f93e40c3340 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f82fd7fa640 1 -- 172.21.15.19:0/1806753867 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f82f807a140 con 0x7f83000a9b00 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f82fd7fa640 10 client.4287.objecter ms_dispatch 0x7f8300000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f82fd7fa640 3 client.4287.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.562+0000 7f82fd7fa640 3 client.4287.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 10 client.4287.objecter checking command 1 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 1 --2- 172.21.15.19:0/1806753867 >> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] conn(0x7f82ec073c50 0x7f82ec076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 20 client.4287.objecter _get_session s=0x7f82ec073580 osd=3 3 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 20 client.4287.objecter put_session s=0x7f82ec073580 osd=3 3 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 15 client.4287.objecter _session_command_op_remove -1 1 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 20 client.4287.objecter _get_session s=0x7f82ec073580 osd=3 3 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 20 client.4287.objecter get_session s=0x7f82ec073580 osd=3 3 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 15 client.4287.objecter _session_command_op_assign 3 1 2024-05-04T17:45:25.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 20 client.4287.objecter put_session s=0x7f82ec073580 osd=3 4 2024-05-04T17:45:25.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 10 client.4287.objecter _send_command 1 2024-05-04T17:45:25.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 1 -- 172.21.15.19:0/1806753867 --> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f82f8038e50 con 0x7f82ec073c50 2024-05-04T17:45:25.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 20 client.4287.objecter dump_active .. 0 homeless 2024-05-04T17:45:25.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 1 -- 172.21.15.19:0/1806753867 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f82f8078340 con 0x7f83000a9b00 2024-05-04T17:45:25.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f82fd7fa640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-05-04T17:45:25.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.563+0000 7f8305364640 1 --2- 172.21.15.19:0/1806753867 >> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] conn(0x7f82ec073c50 0x7f82ec076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.564+0000 7f8305364640 10 monclient: get_auth_request con 0x7f82ec073c50 auth_method 0 2024-05-04T17:45:25.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.564+0000 7f15b8ff9640 1 -- 172.21.15.19:0/3788873225 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f15b401a070 con 0x7f15bc144180 2024-05-04T17:45:25.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.564+0000 7f15b8ff9640 1 --2- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f159c06d810 0x7f159c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.565+0000 7f15bb7fe640 1 --2- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f159c06d810 0x7f159c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.565+0000 7f15bb7fe640 10 monclient: get_auth_request con 0x7f159c06d810 auth_method 0 2024-05-04T17:45:25.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.565+0000 7f15bb7fe640 1 --2- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f159c06d810 0x7f159c06fcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f15ac035000 tx=0x7f15ac02e000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.565+0000 7f15b8ff9640 10 client.4345.objecter ms_handle_connect 0x7f159c06d810 2024-05-04T17:45:25.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.567+0000 7f8305364640 1 --2- 172.21.15.19:0/1806753867 >> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] conn(0x7f82ec073c50 0x7f82ec076070 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.576 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.567+0000 7f82fd7fa640 10 client.4287.objecter ms_handle_connect 0x7f82ec073c50 2024-05-04T17:45:25.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 1 -- 172.21.15.19:0/3788873225 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f15b4056dc0 con 0x7f15bc144180 2024-05-04T17:45:25.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 10 client.4345.objecter ms_dispatch 0x7f15bc000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:25.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 3 client.4345.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:25.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 3 client.4345.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:25.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 10 client.4345.objecter checking command 1 2024-05-04T17:45:25.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 1 --2- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] conn(0x7f159c073c50 0x7f159c076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 20 client.4345.objecter _get_session s=0x7f159c073580 osd=4 3 2024-05-04T17:45:25.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 20 client.4345.objecter put_session s=0x7f159c073580 osd=4 3 2024-05-04T17:45:25.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 15 client.4345.objecter _session_command_op_remove -1 1 2024-05-04T17:45:25.577 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 20 client.4345.objecter _get_session s=0x7f159c073580 osd=4 3 2024-05-04T17:45:25.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 20 client.4345.objecter get_session s=0x7f159c073580 osd=4 3 2024-05-04T17:45:25.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 15 client.4345.objecter _session_command_op_assign 4 1 2024-05-04T17:45:25.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 20 client.4345.objecter put_session s=0x7f159c073580 osd=4 4 2024-05-04T17:45:25.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 10 client.4345.objecter _send_command 1 2024-05-04T17:45:25.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 1 -- 172.21.15.19:0/3788873225 --> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15b4004e50 con 0x7f159c073c50 2024-05-04T17:45:25.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.568+0000 7f15b8ff9640 20 client.4345.objecter dump_active .. 0 homeless 2024-05-04T17:45:25.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 1 -- 172.21.15.19:0/908760950 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c01d070 con 0x7f67341423e0 2024-05-04T17:45:25.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:25.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 10 monclient: dump: 2024-05-04T17:45:25.579 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.579 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.579 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.579 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.579 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.579 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.579 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 1 -- 172.21.15.19:0/908760950 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f672c018440 con 0x7f67341423e0 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 1 -- 172.21.15.19:0/908760950 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c018870 con 0x7f67341423e0 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f671bfff640 10 monclient: dump: 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.580 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.581 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.581 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.581 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.581 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.581 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.581 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f6739d84640 5 monclient: authenticate success, global_id 4348 2024-05-04T17:45:25.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f67317fa640 4 set_mon_vals no callback set 2024-05-04T17:45:25.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f6739d84640 10 monclient: _renew_subs 2024-05-04T17:45:25.581 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f15b2ffd640 1 --2- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] conn(0x7f159c073c50 0x7f159c076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f6739d84640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f6739d84640 1 -- 172.21.15.19:0/908760950 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6734141230 con 0x7f67341423e0 2024-05-04T17:45:25.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f6739d84640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f6739d84640 10 monclient: _renew_subs 2024-05-04T17:45:25.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f6739d84640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f6739d84640 1 -- 172.21.15.19:0/908760950 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67340b40a0 con 0x7f67341423e0 2024-05-04T17:45:25.582 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f6739d84640 1 librados: init done 2024-05-04T17:45:25.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.569+0000 7f15b2ffd640 10 monclient: get_auth_request con 0x7f159c073c50 auth_method 0 2024-05-04T17:45:25.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.570+0000 7f6739d84640 10 client.4348.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.570+0000 7f6739d84640 15 client.4348.objecter _session_command_op_assign -1 1 2024-05-04T17:45:25.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.570+0000 7f6739d84640 20 client.4348.objecter _get_session osd=-1 returning homeless 2024-05-04T17:45:25.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.570+0000 7f6739d84640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.570+0000 7f6739d84640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.583 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.570+0000 7f6739d84640 1 -- 172.21.15.19:0/908760950 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6734141520 con 0x7f67341423e0 2024-05-04T17:45:25.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 1 -- 172.21.15.19:0/1644027812 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c0017070 con 0x7f21c40a9720 2024-05-04T17:45:25.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:25.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 10 monclient: dump: 2024-05-04T17:45:25.584 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.584 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 1 -- 172.21.15.19:0/1644027812 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21c00035f0 con 0x7f21c40a9720 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 1 -- 172.21.15.19:0/1644027812 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c00039e0 con 0x7f21c40a9720 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21baffd640 10 monclient: dump: 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21cc570640 5 monclient: authenticate success, global_id 4351 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21cc570640 10 monclient: _renew_subs 2024-05-04T17:45:25.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21c8b09640 4 set_mon_vals no callback set 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21cc570640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21cc570640 1 -- 172.21.15.19:0/1644027812 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21c4142710 con 0x7f21c40a9720 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21cc570640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21cc570640 10 monclient: _renew_subs 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21cc570640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21cc570640 1 -- 172.21.15.19:0/1644027812 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21c4140950 con 0x7f21c40a9720 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.572+0000 7f21cc570640 1 librados: init done 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.573+0000 7f21cc570640 10 client.4351.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.573+0000 7f21cc570640 15 client.4351.objecter _session_command_op_assign -1 1 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.573+0000 7f21cc570640 20 client.4351.objecter _get_session osd=-1 returning homeless 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.573+0000 7f21cc570640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.573+0000 7f21cc570640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.573+0000 7f21cc570640 1 -- 172.21.15.19:0/1644027812 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f21c406a700 con 0x7f21c40a9720 2024-05-04T17:45:25.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.574+0000 7f15b8ff9640 10 client.4345.objecter ms_handle_connect 0x7f159c073c50 2024-05-04T17:45:25.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.574+0000 7f15b2ffd640 1 --2- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] conn(0x7f159c073c50 0x7f159c076070 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.586+0000 7f02ab7fe640 1 -- 172.21.15.19:0/3067830237 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f02bc00ac30 con 0x7f02c40a9b00 2024-05-04T17:45:25.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.587+0000 7f02ab7fe640 1 --2- 172.21.15.19:0/3067830237 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f02b006d840 0x7f02b006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.587+0000 7f02c2d9d640 1 --2- 172.21.15.19:0/3067830237 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f02b006d840 0x7f02b006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.587+0000 7f02c2d9d640 10 monclient: get_auth_request con 0x7f02b006d840 auth_method 0 2024-05-04T17:45:25.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.587+0000 7f02ab7fe640 10 client.4265.objecter ms_handle_connect 0x7f02b006d840 2024-05-04T17:45:25.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.587+0000 7f02c2d9d640 1 --2- 172.21.15.19:0/3067830237 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f02b006d840 0x7f02b006fd00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f02b4036000 tx=0x7f02b4030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.592+0000 7f7a27fff640 1 -- 172.21.15.19:0/3930593910 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7a2001c070 con 0x7f7a30072390 2024-05-04T17:45:25.593 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f7a27fff640 1 --2- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f7a1006d7c0 0x7f7a1006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f7a2f7fe640 1 --2- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f7a1006d7c0 0x7f7a1006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.594 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f7a2f7fe640 10 monclient: get_auth_request con 0x7f7a1006d7c0 auth_method 0 2024-05-04T17:45:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 1 -- 172.21.15.19:0/3067830237 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f02bc057b40 con 0x7f02c40a9b00 2024-05-04T17:45:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 10 client.4265.objecter ms_dispatch 0x7f02c4000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 3 client.4265.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 3 client.4265.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 10 client.4265.objecter checking command 1 2024-05-04T17:45:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 1 --2- 172.21.15.19:0/3067830237 >> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] conn(0x7f02b0073c80 0x7f02b00760a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 20 client.4265.objecter _get_session s=0x7f02b00735b0 osd=6 3 2024-05-04T17:45:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 20 client.4265.objecter put_session s=0x7f02b00735b0 osd=6 3 2024-05-04T17:45:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 15 client.4265.objecter _session_command_op_remove -1 1 2024-05-04T17:45:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 20 client.4265.objecter _get_session s=0x7f02b00735b0 osd=6 3 2024-05-04T17:45:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 20 client.4265.objecter get_session s=0x7f02b00735b0 osd=6 3 2024-05-04T17:45:25.595 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 15 client.4265.objecter _session_command_op_assign 6 1 2024-05-04T17:45:25.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 20 client.4265.objecter put_session s=0x7f02b00735b0 osd=6 4 2024-05-04T17:45:25.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 10 client.4265.objecter _send_command 1 2024-05-04T17:45:25.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 1 -- 172.21.15.19:0/3067830237 --> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02bc005820 con 0x7f02b0073c80 2024-05-04T17:45:25.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.593+0000 7f02ab7fe640 20 client.4265.objecter dump_active .. 0 homeless 2024-05-04T17:45:25.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f7a2f7fe640 1 --2- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f7a1006d7c0 0x7f7a1006fc80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7a28006fd0 tx=0x7f7a28008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f7a27fff640 10 client.4339.objecter ms_handle_connect 0x7f7a1006d7c0 2024-05-04T17:45:25.596 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f02c359e640 1 --2- 172.21.15.19:0/3067830237 >> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] conn(0x7f02b0073c80 0x7f02b00760a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f775affd640 1 -- 172.21.15.19:0/3138761156 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764017070 con 0x7f776c153790 2024-05-04T17:45:25.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f775affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f775affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:25.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f775affd640 10 monclient: dump: 2024-05-04T17:45:25.597 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f775affd640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f775affd640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f775affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f775affd640 1 -- 172.21.15.19:0/3138761156 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7764003da0 con 0x7f776c153790 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f775affd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.598 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f775affd640 1 -- 172.21.15.19:0/3138761156 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77640052b0 con 0x7f776c153790 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f776b59e640 5 monclient: authenticate success, global_id 4271 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f776b59e640 10 monclient: _renew_subs 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f776b59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.595+0000 7f776b59e640 1 -- 172.21.15.19:0/3138761156 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f776c150b90 con 0x7f776c153790 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.594+0000 7f7768d99640 4 set_mon_vals no callback set 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.595+0000 7f775affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.595+0000 7f775affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.595+0000 7f775affd640 10 monclient: dump: 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.599 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.600 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.600 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.600 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.600 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.595+0000 7f776b59e640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.595+0000 7f776b59e640 10 monclient: _renew_subs 2024-05-04T17:45:25.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.595+0000 7f776b59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.595+0000 7f776b59e640 1 -- 172.21.15.19:0/3138761156 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f776c14fbf0 con 0x7f776c153790 2024-05-04T17:45:25.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.595+0000 7f776b59e640 1 librados: init done 2024-05-04T17:45:25.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.595+0000 7f02c359e640 10 monclient: get_auth_request con 0x7f02b0073c80 auth_method 0 2024-05-04T17:45:25.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f776b59e640 10 client.4271.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f776b59e640 15 client.4271.objecter _session_command_op_assign -1 1 2024-05-04T17:45:25.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f776b59e640 20 client.4271.objecter _get_session osd=-1 returning homeless 2024-05-04T17:45:25.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f776b59e640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f776b59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f776b59e640 1 -- 172.21.15.19:0/3138761156 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f776c151c00 con 0x7f776c153790 2024-05-04T17:45:25.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 1 -- 172.21.15.19:0/3930593910 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f7a20057600 con 0x7f7a30072390 2024-05-04T17:45:25.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 10 client.4339.objecter ms_dispatch 0x7f7a30000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:25.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 3 client.4339.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:25.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 3 client.4339.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:25.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f93ead9d640 1 --2- 172.21.15.19:0/1360227486 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40c3340 0x7f93e40cb480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f93ec075b10 tx=0x7f93ec069180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f93daffd640 10 client.?.objecter ms_handle_connect 0x7f93e40c3340 2024-05-04T17:45:25.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f93daffd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 10 client.4339.objecter checking command 1 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 1 --2- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] conn(0x7f7a10073c00 0x7f7a10076020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 20 client.4339.objecter _get_session s=0x7f7a10073530 osd=5 3 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 20 client.4339.objecter put_session s=0x7f7a10073530 osd=5 3 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 15 client.4339.objecter _session_command_op_remove -1 1 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 20 client.4339.objecter _get_session s=0x7f7a10073530 osd=5 3 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 20 client.4339.objecter get_session s=0x7f7a10073530 osd=5 3 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 15 client.4339.objecter _session_command_op_assign 5 1 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 20 client.4339.objecter put_session s=0x7f7a10073530 osd=5 4 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 10 client.4339.objecter _send_command 1 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 1 -- 172.21.15.19:0/3930593910 --> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7a20022ec0 con 0x7f7a10073c00 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.596+0000 7f7a27fff640 20 client.4339.objecter dump_active .. 0 homeless 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.597+0000 7f7a2ffff640 1 --2- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] conn(0x7f7a10073c00 0x7f7a10076020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.598+0000 7f7a2ffff640 10 monclient: get_auth_request con 0x7f7a10073c00 auth_method 0 2024-05-04T17:45:25.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.599+0000 7f02c359e640 1 --2- 172.21.15.19:0/3067830237 >> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] conn(0x7f02b0073c80 0x7f02b00760a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.599+0000 7f02ab7fe640 10 client.4265.objecter ms_handle_connect 0x7f02b0073c80 2024-05-04T17:45:25.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.601+0000 7f7a27fff640 10 client.4339.objecter ms_handle_connect 0x7f7a10073c00 2024-05-04T17:45:25.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.601+0000 7f7a2ffff640 1 --2- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] conn(0x7f7a10073c00 0x7f7a10076020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.621+0000 7f21baffd640 1 -- 172.21.15.19:0/1644027812 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f21c00113a0 con 0x7f21c40a9720 2024-05-04T17:45:25.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.622+0000 7f21baffd640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f21a806d7c0 0x7f21a806fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.622+0000 7f21c9b0b640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f21a806d7c0 0x7f21a806fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.622+0000 7f21c9b0b640 10 monclient: get_auth_request con 0x7f21a806d7c0 auth_method 0 2024-05-04T17:45:25.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.623+0000 7f21c9b0b640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f21a806d7c0 0x7f21a806fc80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f21c41350b0 tx=0x7f21b4007a40 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.623+0000 7f21baffd640 10 client.4351.objecter ms_handle_connect 0x7f21a806d7c0 2024-05-04T17:45:25.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.625+0000 7f82fd7fa640 1 -- 172.21.15.19:0/1806753867 <== osd.3 v2:172.21.15.19:6819/2537429684 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f82f8038e50 con 0x7f82ec073c50 2024-05-04T17:45:25.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.625+0000 7f82fd7fa640 10 client.4287.objecter ms_dispatch 0x7f8300000b90 command_reply(tid 1: 0 ) 2024-05-04T17:45:25.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.625+0000 7f82fd7fa640 10 client.4287.objecter _finish_command 1 = system:0 2024-05-04T17:45:25.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.625+0000 7f82fd7fa640 20 client.4287.objecter put_session s=0x7f82ec073580 osd=3 4 2024-05-04T17:45:25.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.625+0000 7f82fd7fa640 15 client.4287.objecter _session_command_op_remove 3 1 2024-05-04T17:45:25.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 1 -- 172.21.15.19:0/1644027812 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f21c0056750 con 0x7f21c40a9720 2024-05-04T17:45:25.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 10 client.4351.objecter ms_dispatch 0x7f21c4000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:25.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 3 client.4351.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:25.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 3 client.4351.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:25.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 10 client.4351.objecter checking command 1 2024-05-04T17:45:25.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] conn(0x7f21a8073c00 0x7f21a8076020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 20 client.4351.objecter _get_session s=0x7f21a8073530 osd=0 3 2024-05-04T17:45:25.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 20 client.4351.objecter put_session s=0x7f21a8073530 osd=0 3 2024-05-04T17:45:25.637 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 15 client.4351.objecter _session_command_op_remove -1 1 2024-05-04T17:45:25.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 20 client.4351.objecter _get_session s=0x7f21a8073530 osd=0 3 2024-05-04T17:45:25.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 20 client.4351.objecter get_session s=0x7f21a8073530 osd=0 3 2024-05-04T17:45:25.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 15 client.4351.objecter _session_command_op_assign 0 1 2024-05-04T17:45:25.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 20 client.4351.objecter put_session s=0x7f21a8073530 osd=0 4 2024-05-04T17:45:25.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 10 client.4351.objecter _send_command 1 2024-05-04T17:45:25.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 1 -- 172.21.15.19:0/1644027812 --> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f21a80766d0 con 0x7f21a8073c00 2024-05-04T17:45:25.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.636+0000 7f21baffd640 20 client.4351.objecter dump_active .. 0 homeless 2024-05-04T17:45:25.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.637+0000 7f21cab0d640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] conn(0x7f21a8073c00 0x7f21a8076020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.638 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.637+0000 7f21cab0d640 10 monclient: get_auth_request con 0x7f21a8073c00 auth_method 0 2024-05-04T17:45:25.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.640+0000 7f8306dc7640 10 client.4287.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-04T17:45:25.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.640+0000 7f8306dc7640 15 client.4287.objecter _session_command_op_assign -1 2 2024-05-04T17:45:25.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.640+0000 7f8306dc7640 20 client.4287.objecter _get_session s=0x7f82ec073580 osd=3 3 2024-05-04T17:45:25.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.640+0000 7f8306dc7640 20 client.4287.objecter put_session s=0x7f82ec073580 osd=3 3 2024-05-04T17:45:25.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.640+0000 7f8306dc7640 20 client.4287.objecter _get_session s=0x7f82ec073580 osd=3 3 2024-05-04T17:45:25.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.640+0000 7f8306dc7640 15 client.4287.objecter _session_command_op_remove -1 2 2024-05-04T17:45:25.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.640+0000 7f8306dc7640 20 client.4287.objecter get_session s=0x7f82ec073580 osd=3 3 2024-05-04T17:45:25.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.640+0000 7f8306dc7640 15 client.4287.objecter _session_command_op_assign 3 2 2024-05-04T17:45:25.641 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.640+0000 7f8306dc7640 20 client.4287.objecter put_session s=0x7f82ec073580 osd=3 4 2024-05-04T17:45:25.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.640+0000 7f8306dc7640 10 client.4287.objecter _send_command 2 2024-05-04T17:45:25.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.640+0000 7f8306dc7640 1 -- 172.21.15.19:0/1806753867 --> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f830006a700 con 0x7f82ec073c50 2024-05-04T17:45:25.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.642+0000 7f21cab0d640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] conn(0x7f21a8073c00 0x7f21a8076020 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.643 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.642+0000 7f21baffd640 10 client.4351.objecter ms_handle_connect 0x7f21a8073c00 2024-05-04T17:45:25.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.647+0000 7f15b8ff9640 1 -- 172.21.15.19:0/3788873225 <== osd.4 v2:172.21.15.186:6818/356995042 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15b4004e50 con 0x7f159c073c50 2024-05-04T17:45:25.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.647+0000 7f15b8ff9640 10 client.4345.objecter ms_dispatch 0x7f15bc000b90 command_reply(tid 1: 0 ) 2024-05-04T17:45:25.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.647+0000 7f15b8ff9640 10 client.4345.objecter _finish_command 1 = system:0 2024-05-04T17:45:25.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.647+0000 7f15b8ff9640 20 client.4345.objecter put_session s=0x7f159c073580 osd=4 4 2024-05-04T17:45:25.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.647+0000 7f15b8ff9640 15 client.4345.objecter _session_command_op_remove 4 1 2024-05-04T17:45:25.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.648+0000 7f82fd7fa640 1 -- 172.21.15.19:0/1806753867 <== osd.3 v2:172.21.15.19:6819/2537429684 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f830006a700 con 0x7f82ec073c50 2024-05-04T17:45:25.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.648+0000 7f82fd7fa640 10 client.4287.objecter ms_dispatch 0x7f8300000b90 command_reply(tid 2: 0 ) 2024-05-04T17:45:25.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.648+0000 7f82fd7fa640 10 client.4287.objecter _finish_command 2 = system:0 2024-05-04T17:45:25.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.648+0000 7f82fd7fa640 20 client.4287.objecter put_session s=0x7f82ec073580 osd=3 4 2024-05-04T17:45:25.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.648+0000 7f82fd7fa640 15 client.4287.objecter _session_command_op_remove 3 2 2024-05-04T17:45:25.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 10 librados: watch_flush enter 2024-05-04T17:45:25.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 10 librados: watch_flush exit 2024-05-04T17:45:25.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 10 client.4287.objecter close_session for osd.3 2024-05-04T17:45:25.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 1 -- 172.21.15.19:0/1806753867 >> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] conn(0x7f82ec073c50 msgr2=0x7f82ec076070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.650 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 1 --2- 172.21.15.19:0/1806753867 >> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] conn(0x7f82ec073c50 0x7f82ec076070 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 20 client.4287.objecter put_session s=0x7f82ec073580 osd=3 1 2024-05-04T17:45:25.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 20 client.4287.objecter shutdown clearing up homeless session... 2024-05-04T17:45:25.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 10 client.4287.objecter successfully canceled tick 2024-05-04T17:45:25.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.650+0000 7f671bfff640 1 -- 172.21.15.19:0/908760950 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f672c020070 con 0x7f67341423e0 2024-05-04T17:45:25.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 1 -- 172.21.15.19:0/1806753867 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f82ec06d810 msgr2=0x7f82ec06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 1 --2- 172.21.15.19:0/1806753867 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f82ec06d810 0x7f82ec06fcd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f82f0006fd0 tx=0x7f82f0009060 comp rx=0 tx=0).stop 2024-05-04T17:45:25.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 10 monclient: shutdown 2024-05-04T17:45:25.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 1 -- 172.21.15.19:0/1806753867 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83000a9b00 msgr2=0x7f8300147fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8306dc7640 1 --2- 172.21.15.19:0/1806753867 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f83000a9b00 0x7f8300147fb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f82f8036040 tx=0x7f82f8004a40 comp rx=0 tx=0).stop 2024-05-04T17:45:25.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.649+0000 7f8305364640 1 -- 172.21.15.19:0/1806753867 reap_dead start 2024-05-04T17:45:25.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.650+0000 7f671bfff640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f672006d7c0 0x7f672006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.650+0000 7f8306dc7640 1 -- 172.21.15.19:0/1806753867 shutdown_connections 2024-05-04T17:45:25.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.650+0000 7f8306dc7640 1 -- 172.21.15.19:0/1806753867 >> 172.21.15.19:0/1806753867 conn(0x7f8300067780 msgr2=0x7f830013f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.650+0000 7f8306dc7640 1 -- 172.21.15.19:0/1806753867 shutdown_connections 2024-05-04T17:45:25.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.650+0000 7f67337fe640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f672006d7c0 0x7f672006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.650+0000 7f67337fe640 10 monclient: get_auth_request con 0x7f672006d7c0 auth_method 0 2024-05-04T17:45:25.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.651+0000 7f8306dc7640 1 -- 172.21.15.19:0/1806753867 wait complete. 2024-05-04T17:45:25.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.651+0000 7f8306dc7640 1 librados: shutdown 2024-05-04T17:45:25.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.651+0000 7f671bfff640 10 client.4348.objecter ms_handle_connect 0x7f672006d7c0 2024-05-04T17:45:25.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.651+0000 7f67337fe640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f672006d7c0 0x7f672006fc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6724009710 tx=0x7f6724006c90 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 1 -- 172.21.15.19:0/908760950 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f672c05ba30 con 0x7f67341423e0 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 10 client.4348.objecter ms_dispatch 0x7f6734000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 3 client.4348.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 3 client.4348.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 10 client.4348.objecter checking command 1 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] conn(0x7f6720073c00 0x7f6720076020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 20 client.4348.objecter _get_session s=0x7f6720073530 osd=1 3 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 20 client.4348.objecter put_session s=0x7f6720073530 osd=1 3 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 15 client.4348.objecter _session_command_op_remove -1 1 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 20 client.4348.objecter _get_session s=0x7f6720073530 osd=1 3 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 20 client.4348.objecter get_session s=0x7f6720073530 osd=1 3 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 15 client.4348.objecter _session_command_op_assign 1 1 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 20 client.4348.objecter put_session s=0x7f6720073530 osd=1 4 2024-05-04T17:45:25.655 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 10 client.4348.objecter _send_command 1 2024-05-04T17:45:25.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 1 -- 172.21.15.19:0/908760950 --> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67200766d0 con 0x7f6720073c00 2024-05-04T17:45:25.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.654+0000 7f671bfff640 20 client.4348.objecter dump_active .. 0 homeless 2024-05-04T17:45:25.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.655+0000 7f6732ffd640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] conn(0x7f6720073c00 0x7f6720076020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.655+0000 7f6732ffd640 10 monclient: get_auth_request con 0x7f6720073c00 auth_method 0 2024-05-04T17:45:25.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.660+0000 7f671bfff640 10 client.4348.objecter ms_handle_connect 0x7f6720073c00 2024-05-04T17:45:25.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.660+0000 7f6732ffd640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] conn(0x7f6720073c00 0x7f6720076020 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.661+0000 7f7a27fff640 1 -- 172.21.15.19:0/3930593910 <== osd.5 v2:172.21.15.186:6810/3728813703 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7a20022ec0 con 0x7f7a10073c00 2024-05-04T17:45:25.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.661+0000 7f7a27fff640 10 client.4339.objecter ms_dispatch 0x7f7a30000b90 command_reply(tid 1: 0 ) 2024-05-04T17:45:25.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.661+0000 7f7a27fff640 10 client.4339.objecter _finish_command 1 = system:0 2024-05-04T17:45:25.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.661+0000 7f7a27fff640 20 client.4339.objecter put_session s=0x7f7a10073530 osd=5 4 2024-05-04T17:45:25.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.661+0000 7f7a27fff640 15 client.4339.objecter _session_command_op_remove 5 1 2024-05-04T17:45:25.665 INFO:teuthology.orchestra.run.smithi019.stdout:68719476744 2024-05-04T17:45:25.665 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-04T17:45:25.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93daffd640 1 -- 172.21.15.19:0/1360227486 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ec0bf070 con 0x7f93e40c3340 2024-05-04T17:45:25.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:25.671 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93daffd640 10 monclient: dump: 2024-05-04T17:45:25.671 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.671 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.671 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.671 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.671 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.671 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.671 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.671 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93daffd640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93daffd640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93eb59e640 5 monclient: authenticate success, global_id 4274 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93eb59e640 10 monclient: _renew_subs 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93eb59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93eb59e640 1 -- 172.21.15.19:0/1360227486 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e40cdae0 con 0x7f93e40c3340 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93eb59e640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93eb59e640 10 monclient: _renew_subs 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93eb59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93eb59e640 1 -- 172.21.15.19:0/1360227486 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e40cc270 con 0x7f93e40c3340 2024-05-04T17:45:25.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.670+0000 7f93eb59e640 1 librados: init done 2024-05-04T17:45:25.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.671+0000 7f93eb59e640 10 client.4274.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.671+0000 7f93eb59e640 15 client.4274.objecter _session_command_op_assign -1 1 2024-05-04T17:45:25.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.671+0000 7f93eb59e640 20 client.4274.objecter _get_session osd=-1 returning homeless 2024-05-04T17:45:25.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.671+0000 7f93eb59e640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.671+0000 7f93eb59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.671+0000 7f93eb59e640 1 -- 172.21.15.19:0/1360227486 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f93e4005710 con 0x7f93e40c3340 2024-05-04T17:45:25.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.678+0000 7f15c1e3e640 10 client.4345.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-04T17:45:25.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.678+0000 7f15c1e3e640 15 client.4345.objecter _session_command_op_assign -1 2 2024-05-04T17:45:25.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.678+0000 7f15c1e3e640 20 client.4345.objecter _get_session s=0x7f159c073580 osd=4 3 2024-05-04T17:45:25.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.678+0000 7f15c1e3e640 20 client.4345.objecter put_session s=0x7f159c073580 osd=4 3 2024-05-04T17:45:25.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.678+0000 7f15c1e3e640 20 client.4345.objecter _get_session s=0x7f159c073580 osd=4 3 2024-05-04T17:45:25.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.678+0000 7f15c1e3e640 15 client.4345.objecter _session_command_op_remove -1 2 2024-05-04T17:45:25.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.678+0000 7f15c1e3e640 20 client.4345.objecter get_session s=0x7f159c073580 osd=4 3 2024-05-04T17:45:25.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.678+0000 7f15c1e3e640 15 client.4345.objecter _session_command_op_assign 4 2 2024-05-04T17:45:25.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.678+0000 7f15c1e3e640 20 client.4345.objecter put_session s=0x7f159c073580 osd=4 4 2024-05-04T17:45:25.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.678+0000 7f15c1e3e640 10 client.4345.objecter _send_command 2 2024-05-04T17:45:25.680 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.678+0000 7f15c1e3e640 1 -- 172.21.15.19:0/3788873225 --> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f15a0001150 con 0x7f159c073c50 2024-05-04T17:45:25.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.684+0000 7f7a35af2640 10 client.4339.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-04T17:45:25.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.684+0000 7f7a35af2640 15 client.4339.objecter _session_command_op_assign -1 2 2024-05-04T17:45:25.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.684+0000 7f7a35af2640 20 client.4339.objecter _get_session s=0x7f7a10073530 osd=5 3 2024-05-04T17:45:25.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.684+0000 7f7a35af2640 20 client.4339.objecter put_session s=0x7f7a10073530 osd=5 3 2024-05-04T17:45:25.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.684+0000 7f7a35af2640 20 client.4339.objecter _get_session s=0x7f7a10073530 osd=5 3 2024-05-04T17:45:25.685 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.684+0000 7f7a35af2640 15 client.4339.objecter _session_command_op_remove -1 2 2024-05-04T17:45:25.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.684+0000 7f7a35af2640 20 client.4339.objecter get_session s=0x7f7a10073530 osd=5 3 2024-05-04T17:45:25.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.684+0000 7f7a35af2640 15 client.4339.objecter _session_command_op_assign 5 2 2024-05-04T17:45:25.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.684+0000 7f7a35af2640 20 client.4339.objecter put_session s=0x7f7a10073530 osd=5 4 2024-05-04T17:45:25.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.684+0000 7f7a35af2640 10 client.4339.objecter _send_command 2 2024-05-04T17:45:25.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.684+0000 7f7a35af2640 1 -- 172.21.15.19:0/3930593910 --> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7a30068b10 con 0x7f7a10073c00 2024-05-04T17:45:25.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.685+0000 7f02ab7fe640 1 -- 172.21.15.19:0/3067830237 <== osd.6 v2:172.21.15.186:6802/4252462313 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f02bc005820 con 0x7f02b0073c80 2024-05-04T17:45:25.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.685+0000 7f02ab7fe640 10 client.4265.objecter ms_dispatch 0x7f02c4000b90 command_reply(tid 1: 0 ) 2024-05-04T17:45:25.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.685+0000 7f02ab7fe640 10 client.4265.objecter _finish_command 1 = system:0 2024-05-04T17:45:25.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.685+0000 7f02ab7fe640 20 client.4265.objecter put_session s=0x7f02b00735b0 osd=6 4 2024-05-04T17:45:25.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.685+0000 7f02ab7fe640 15 client.4265.objecter _session_command_op_remove 6 1 2024-05-04T17:45:25.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.690+0000 7f15b8ff9640 1 -- 172.21.15.19:0/3788873225 <== osd.4 v2:172.21.15.186:6818/356995042 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f15a0001150 con 0x7f159c073c50 2024-05-04T17:45:25.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.690+0000 7f15b8ff9640 10 client.4345.objecter ms_dispatch 0x7f15bc000b90 command_reply(tid 2: 0 ) 2024-05-04T17:45:25.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.690+0000 7f15b8ff9640 10 client.4345.objecter _finish_command 2 = system:0 2024-05-04T17:45:25.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.690+0000 7f15b8ff9640 20 client.4345.objecter put_session s=0x7f159c073580 osd=4 4 2024-05-04T17:45:25.691 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.690+0000 7f15b8ff9640 15 client.4345.objecter _session_command_op_remove 4 2 2024-05-04T17:45:25.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 10 librados: watch_flush enter 2024-05-04T17:45:25.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 10 librados: watch_flush exit 2024-05-04T17:45:25.692 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 10 client.4345.objecter close_session for osd.4 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 1 -- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] conn(0x7f159c073c50 msgr2=0x7f159c076070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 1 --2- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] conn(0x7f159c073c50 0x7f159c076070 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 20 client.4345.objecter put_session s=0x7f159c073580 osd=4 1 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 20 client.4345.objecter shutdown clearing up homeless session... 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 10 client.4345.objecter successfully canceled tick 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 1 -- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f159c06d810 msgr2=0x7f159c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 1 --2- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f159c06d810 0x7f159c06fcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f15ac035000 tx=0x7f15ac02e000 comp rx=0 tx=0).stop 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 10 monclient: shutdown 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 20 monclient: shutdown canceling and discarding version request 1 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 1 -- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15bc144180 msgr2=0x7f15bc12b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 1 --2- 172.21.15.19:0/3788873225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15bc144180 0x7f15bc12b0e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f15b400c390 tx=0x7f15b400c850 comp rx=0 tx=0).stop 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15bbfff640 1 -- 172.21.15.19:0/3788873225 reap_dead start 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 1 -- 172.21.15.19:0/3788873225 shutdown_connections 2024-05-04T17:45:25.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 1 -- 172.21.15.19:0/3788873225 >> 172.21.15.19:0/3788873225 conn(0x7f15bc0b4580 msgr2=0x7f15bc0b1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 1 -- 172.21.15.19:0/3788873225 shutdown_connections 2024-05-04T17:45:25.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 1 -- 172.21.15.19:0/3788873225 wait complete. 2024-05-04T17:45:25.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.691+0000 7f15c1e3e640 1 librados: shutdown 2024-05-04T17:45:25.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.692+0000 7f7a27fff640 1 -- 172.21.15.19:0/3930593910 <== osd.5 v2:172.21.15.186:6810/3728813703 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7a30068b10 con 0x7f7a10073c00 2024-05-04T17:45:25.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.692+0000 7f7a27fff640 10 client.4339.objecter ms_dispatch 0x7f7a30000b90 command_reply(tid 2: 0 ) 2024-05-04T17:45:25.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.692+0000 7f7a27fff640 10 client.4339.objecter _finish_command 2 = system:0 2024-05-04T17:45:25.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.692+0000 7f7a27fff640 20 client.4339.objecter put_session s=0x7f7a10073530 osd=5 4 2024-05-04T17:45:25.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.692+0000 7f7a27fff640 15 client.4339.objecter _session_command_op_remove 5 2 2024-05-04T17:45:25.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 10 librados: watch_flush enter 2024-05-04T17:45:25.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 10 librados: watch_flush exit 2024-05-04T17:45:25.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 10 client.4339.objecter close_session for osd.5 2024-05-04T17:45:25.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 1 -- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] conn(0x7f7a10073c00 msgr2=0x7f7a10076020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 1 --2- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] conn(0x7f7a10073c00 0x7f7a10076020 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 20 client.4339.objecter put_session s=0x7f7a10073530 osd=5 1 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 20 client.4339.objecter shutdown clearing up homeless session... 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 10 client.4339.objecter successfully canceled tick 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 1 -- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f7a1006d7c0 msgr2=0x7f7a1006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 1 --2- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f7a1006d7c0 0x7f7a1006fc80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7a28006fd0 tx=0x7f7a28008040 comp rx=0 tx=0).stop 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 10 monclient: shutdown 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 20 monclient: shutdown canceling and discarding version request 1 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 1 -- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a30072390 msgr2=0x7f7a300b3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 1 --2- 172.21.15.19:0/3930593910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a30072390 0x7f7a300b3070 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7a2000b9c0 tx=0x7f7a2000be80 comp rx=0 tx=0).stop 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a2ffff640 1 -- 172.21.15.19:0/3930593910 reap_dead start 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 1 -- 172.21.15.19:0/3930593910 shutdown_connections 2024-05-04T17:45:25.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.693+0000 7f7a35af2640 1 -- 172.21.15.19:0/3930593910 >> 172.21.15.19:0/3930593910 conn(0x7f7a300b4580 msgr2=0x7f7a301478d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.696+0000 7f7a35af2640 1 -- 172.21.15.19:0/3930593910 shutdown_connections 2024-05-04T17:45:25.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.700+0000 7f7a35af2640 1 -- 172.21.15.19:0/3930593910 wait complete. 2024-05-04T17:45:25.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.700+0000 7f7a35af2640 1 librados: shutdown 2024-05-04T17:45:25.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.702+0000 7f02c9175640 10 client.4265.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-04T17:45:25.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.702+0000 7f02c9175640 15 client.4265.objecter _session_command_op_assign -1 2 2024-05-04T17:45:25.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.702+0000 7f02c9175640 20 client.4265.objecter _get_session s=0x7f02b00735b0 osd=6 3 2024-05-04T17:45:25.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.702+0000 7f02c9175640 20 client.4265.objecter put_session s=0x7f02b00735b0 osd=6 3 2024-05-04T17:45:25.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.702+0000 7f02c9175640 20 client.4265.objecter _get_session s=0x7f02b00735b0 osd=6 3 2024-05-04T17:45:25.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.702+0000 7f02c9175640 15 client.4265.objecter _session_command_op_remove -1 2 2024-05-04T17:45:25.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.702+0000 7f02c9175640 20 client.4265.objecter get_session s=0x7f02b00735b0 osd=6 3 2024-05-04T17:45:25.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.702+0000 7f02c9175640 15 client.4265.objecter _session_command_op_assign 6 2 2024-05-04T17:45:25.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.702+0000 7f02c9175640 20 client.4265.objecter put_session s=0x7f02b00735b0 osd=6 4 2024-05-04T17:45:25.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.702+0000 7f02c9175640 10 client.4265.objecter _send_command 2 2024-05-04T17:45:25.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.702+0000 7f02c9175640 1 -- 172.21.15.19:0/3067830237 --> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f02c406a700 con 0x7f02b0073c80 2024-05-04T17:45:25.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.703+0000 7f775affd640 1 -- 172.21.15.19:0/3138761156 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f776401a070 con 0x7f776c153790 2024-05-04T17:45:25.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.703+0000 7f775affd640 1 --2- 172.21.15.19:0/3138761156 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f774c06d7c0 0x7f774c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.704+0000 7f776a59c640 1 --2- 172.21.15.19:0/3138761156 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f774c06d7c0 0x7f774c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.704+0000 7f776a59c640 10 monclient: get_auth_request con 0x7f774c06d7c0 auth_method 0 2024-05-04T17:45:25.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.704+0000 7f776a59c640 1 --2- 172.21.15.19:0/3138761156 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f774c06d7c0 0x7f774c06fc80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f775c0047c0 tx=0x7f775c009120 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.705+0000 7f775affd640 10 client.4271.objecter ms_handle_connect 0x7f774c06d7c0 2024-05-04T17:45:25.707 INFO:teuthology.orchestra.run.smithi019.stdout:81604378633 2024-05-04T17:45:25.707 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-04T17:45:25.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.713+0000 7f02ab7fe640 1 -- 172.21.15.19:0/3067830237 <== osd.6 v2:172.21.15.186:6802/4252462313 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f02c406a700 con 0x7f02b0073c80 2024-05-04T17:45:25.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.713+0000 7f02ab7fe640 10 client.4265.objecter ms_dispatch 0x7f02c4000b90 command_reply(tid 2: 0 ) 2024-05-04T17:45:25.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.713+0000 7f02ab7fe640 10 client.4265.objecter _finish_command 2 = system:0 2024-05-04T17:45:25.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.713+0000 7f02ab7fe640 20 client.4265.objecter put_session s=0x7f02b00735b0 osd=6 4 2024-05-04T17:45:25.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.713+0000 7f02ab7fe640 15 client.4265.objecter _session_command_op_remove 6 2 2024-05-04T17:45:25.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.714+0000 7f02a97fa640 10 librados: watch_flush enter 2024-05-04T17:45:25.715 INFO:teuthology.orchestra.run.smithi019.stdout:55834574854 2024-05-04T17:45:25.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.714+0000 7f02a97fa640 10 librados: watch_flush exit 2024-05-04T17:45:25.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.714+0000 7f02a97fa640 10 client.4265.objecter close_session for osd.6 2024-05-04T17:45:25.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.714+0000 7f02a97fa640 1 -- 172.21.15.19:0/3067830237 >> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] conn(0x7f02b0073c80 msgr2=0x7f02b00760a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.714+0000 7f02a97fa640 1 --2- 172.21.15.19:0/3067830237 >> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] conn(0x7f02b0073c80 0x7f02b00760a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.714+0000 7f02a97fa640 20 client.4265.objecter put_session s=0x7f02b00735b0 osd=6 1 2024-05-04T17:45:25.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.715+0000 7f02a97fa640 20 client.4265.objecter shutdown clearing up homeless session... 2024-05-04T17:45:25.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.715+0000 7f02a97fa640 10 client.4265.objecter successfully canceled tick 2024-05-04T17:45:25.717 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.715+0000 7f02a97fa640 1 -- 172.21.15.19:0/3067830237 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f02b006d840 msgr2=0x7f02b006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.715+0000 7f02a97fa640 1 --2- 172.21.15.19:0/3067830237 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f02b006d840 0x7f02b006fd00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f02b4036000 tx=0x7f02b4030000 comp rx=0 tx=0).stop 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.715+0000 7f02a97fa640 10 monclient: shutdown 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.715+0000 7f02a97fa640 20 monclient: shutdown canceling and discarding version request 1 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.715+0000 7f02a97fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.715+0000 7f02a97fa640 1 -- 172.21.15.19:0/3067830237 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f02c40a9b00 msgr2=0x7f02c41d7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.715+0000 7f02a97fa640 1 --2- 172.21.15.19:0/3067830237 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f02c40a9b00 0x7f02c41d7270 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00c8a0 tx=0x7f02bc00cd60 comp rx=0 tx=0).stop 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.715+0000 7f02c359e640 1 -- 172.21.15.19:0/3067830237 reap_dead start 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.716+0000 7f02a97fa640 1 -- 172.21.15.19:0/3067830237 shutdown_connections 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.716+0000 7f02a97fa640 1 -- 172.21.15.19:0/3067830237 >> 172.21.15.19:0/3067830237 conn(0x7f02c4067780 msgr2=0x7f02c40b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.716+0000 7f02a97fa640 1 -- 172.21.15.19:0/3067830237 shutdown_connections 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.716+0000 7f02a97fa640 1 -- 172.21.15.19:0/3067830237 wait complete. 2024-05-04T17:45:25.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.716+0000 7f02a97fa640 1 librados: shutdown 2024-05-04T17:45:25.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.718+0000 7f21baffd640 1 -- 172.21.15.19:0/1644027812 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f21c0054950 con 0x7f21c40a9720 2024-05-04T17:45:25.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.718+0000 7f21baffd640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-05-04T17:45:25.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.718+0000 7f671bfff640 1 -- 172.21.15.19:0/908760950 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f672c059c30 con 0x7f67341423e0 2024-05-04T17:45:25.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.718+0000 7f671bfff640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-05-04T17:45:25.734 INFO:teuthology.orchestra.run.smithi019.stdout:81604378632 2024-05-04T17:45:25.734 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-04T17:45:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.739+0000 7f21baffd640 1 -- 172.21.15.19:0/1644027812 <== osd.0 v2:172.21.15.19:6808/2236557705 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f21a80766d0 con 0x7f21a8073c00 2024-05-04T17:45:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.739+0000 7f21baffd640 10 client.4351.objecter ms_dispatch 0x7f21c4000b90 command_reply(tid 1: 0 ) 2024-05-04T17:45:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.739+0000 7f21baffd640 10 client.4351.objecter _finish_command 1 = system:0 2024-05-04T17:45:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.739+0000 7f21baffd640 20 client.4351.objecter put_session s=0x7f21a8073530 osd=0 4 2024-05-04T17:45:25.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.739+0000 7f21baffd640 15 client.4351.objecter _session_command_op_remove 0 1 2024-05-04T17:45:25.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.752+0000 7f93daffd640 1 -- 172.21.15.19:0/1360227486 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93e4005710 con 0x7f93e40c3340 2024-05-04T17:45:25.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.752+0000 7f93daffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.752+0000 7f93daffd640 1 -- 172.21.15.19:0/1360227486 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ec0ba630 con 0x7f93e40c3340 2024-05-04T17:45:25.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.752+0000 7f93daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.752+0000 7f93daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:25.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.752+0000 7f93daffd640 10 monclient: dump: 2024-05-04T17:45:25.754 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.754 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.754 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.754 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.755 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.755 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.755 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.755 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.755 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.755 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.752+0000 7f93e8d99640 4 set_mon_vals no callback set 2024-05-04T17:45:25.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.752+0000 7f93daffd640 1 -- 172.21.15.19:0/1360227486 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f93ec0c2070 con 0x7f93e40c3340 2024-05-04T17:45:25.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f93daffd640 1 --2- 172.21.15.19:0/1360227486 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f93cc06d770 0x7f93cc06fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.755 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 1 -- 172.21.15.19:0/3138761156 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f7764056fa0 con 0x7f776c153790 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 10 client.4271.objecter ms_dispatch 0x7f776c000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 3 client.4271.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 3 client.4271.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 10 client.4271.objecter checking command 1 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 1 --2- 172.21.15.19:0/3138761156 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x7f774c073c00 0x7f774c076020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 20 client.4271.objecter _get_session s=0x7f774c073530 osd=7 3 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 20 client.4271.objecter put_session s=0x7f774c073530 osd=7 3 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 15 client.4271.objecter _session_command_op_remove -1 1 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 20 client.4271.objecter _get_session s=0x7f774c073530 osd=7 3 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 20 client.4271.objecter get_session s=0x7f774c073530 osd=7 3 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 15 client.4271.objecter _session_command_op_assign 7 1 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 20 client.4271.objecter put_session s=0x7f774c073530 osd=7 4 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 10 client.4271.objecter _send_command 1 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 1 -- 172.21.15.19:0/3138761156 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7764011e50 con 0x7f774c073c00 2024-05-04T17:45:25.756 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 20 client.4271.objecter dump_active .. 0 homeless 2024-05-04T17:45:25.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 1 -- 172.21.15.19:0/3138761156 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f77640551a0 con 0x7f776c153790 2024-05-04T17:45:25.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f775affd640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-05-04T17:45:25.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f93ea59c640 1 --2- 172.21.15.19:0/1360227486 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f93cc06d770 0x7f93cc06fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.753+0000 7f93ea59c640 10 monclient: get_auth_request con 0x7f93cc06d770 auth_method 0 2024-05-04T17:45:25.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.754+0000 7f7769d9b640 1 --2- 172.21.15.19:0/3138761156 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x7f774c073c00 0x7f774c076020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.757 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.754+0000 7f7769d9b640 10 monclient: get_auth_request con 0x7f774c073c00 auth_method 0 2024-05-04T17:45:25.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.754+0000 7f671bfff640 1 -- 172.21.15.19:0/908760950 <== osd.1 v2:172.21.15.19:6800/3360656308 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67200766d0 con 0x7f6720073c00 2024-05-04T17:45:25.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.754+0000 7f671bfff640 10 client.4348.objecter ms_dispatch 0x7f6734000b90 command_reply(tid 1: 0 ) 2024-05-04T17:45:25.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.754+0000 7f671bfff640 10 client.4348.objecter _finish_command 1 = system:0 2024-05-04T17:45:25.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.754+0000 7f671bfff640 20 client.4348.objecter put_session s=0x7f6720073530 osd=1 4 2024-05-04T17:45:25.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.754+0000 7f671bfff640 15 client.4348.objecter _session_command_op_remove 1 1 2024-05-04T17:45:25.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.754+0000 7f93ea59c640 1 --2- 172.21.15.19:0/1360227486 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f93cc06d770 0x7f93cc06fc30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f93dc001560 tx=0x7f93dc007990 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.754+0000 7f93daffd640 10 client.4274.objecter ms_handle_connect 0x7f93cc06d770 2024-05-04T17:45:25.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.758+0000 7f775affd640 10 client.4271.objecter ms_handle_connect 0x7f774c073c00 2024-05-04T17:45:25.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.758+0000 7f7769d9b640 1 --2- 172.21.15.19:0/3138761156 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x7f774c073c00 0x7f774c076020 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 1 -- 172.21.15.19:0/1360227486 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f93ec0f3c40 con 0x7f93e40c3340 2024-05-04T17:45:25.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 10 client.4274.objecter ms_dispatch 0x7f93ec000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 3 client.4274.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 3 client.4274.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 10 client.4274.objecter checking command 1 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 1 --2- 172.21.15.19:0/1360227486 >> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] conn(0x7f93cc073bb0 0x7f93cc075fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 20 client.4274.objecter _get_session s=0x7f93cc0734e0 osd=2 3 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 20 client.4274.objecter put_session s=0x7f93cc0734e0 osd=2 3 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 15 client.4274.objecter _session_command_op_remove -1 1 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 20 client.4274.objecter _get_session s=0x7f93cc0734e0 osd=2 3 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 20 client.4274.objecter get_session s=0x7f93cc0734e0 osd=2 3 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 15 client.4274.objecter _session_command_op_assign 2 1 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 20 client.4274.objecter put_session s=0x7f93cc0734e0 osd=2 4 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 10 client.4274.objecter _send_command 1 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 1 -- 172.21.15.19:0/1360227486 --> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93ec0aeec0 con 0x7f93cc073bb0 2024-05-04T17:45:25.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 20 client.4274.objecter dump_active .. 0 homeless 2024-05-04T17:45:25.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 1 -- 172.21.15.19:0/1360227486 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f93ec0f1e40 con 0x7f93e40c3340 2024-05-04T17:45:25.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.762+0000 7f93daffd640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-05-04T17:45:25.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.763+0000 7f93e9d9b640 1 --2- 172.21.15.19:0/1360227486 >> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] conn(0x7f93cc073bb0 0x7f93cc075fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.764+0000 7f93e9d9b640 10 monclient: get_auth_request con 0x7f93cc073bb0 auth_method 0 2024-05-04T17:45:25.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.766+0000 7f21cc570640 10 client.4351.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-04T17:45:25.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.766+0000 7f21cc570640 15 client.4351.objecter _session_command_op_assign -1 2 2024-05-04T17:45:25.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.766+0000 7f21cc570640 20 client.4351.objecter _get_session s=0x7f21a8073530 osd=0 3 2024-05-04T17:45:25.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.766+0000 7f21cc570640 20 client.4351.objecter put_session s=0x7f21a8073530 osd=0 3 2024-05-04T17:45:25.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.766+0000 7f21cc570640 20 client.4351.objecter _get_session s=0x7f21a8073530 osd=0 3 2024-05-04T17:45:25.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.766+0000 7f21cc570640 15 client.4351.objecter _session_command_op_remove -1 2 2024-05-04T17:45:25.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.766+0000 7f21cc570640 20 client.4351.objecter get_session s=0x7f21a8073530 osd=0 3 2024-05-04T17:45:25.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.766+0000 7f21cc570640 15 client.4351.objecter _session_command_op_assign 0 2 2024-05-04T17:45:25.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.766+0000 7f21cc570640 20 client.4351.objecter put_session s=0x7f21a8073530 osd=0 4 2024-05-04T17:45:25.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.766+0000 7f21cc570640 10 client.4351.objecter _send_command 2 2024-05-04T17:45:25.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.766+0000 7f21cc570640 1 -- 172.21.15.19:0/1644027812 --> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f21c40b2370 con 0x7f21a8073c00 2024-05-04T17:45:25.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.769+0000 7f93e9d9b640 1 --2- 172.21.15.19:0/1360227486 >> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] conn(0x7f93cc073bb0 0x7f93cc075fd0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.770 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.769+0000 7f93daffd640 10 client.4274.objecter ms_handle_connect 0x7f93cc073bb0 2024-05-04T17:45:25.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.777+0000 7f6739d84640 10 client.4348.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-04T17:45:25.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.777+0000 7f6739d84640 15 client.4348.objecter _session_command_op_assign -1 2 2024-05-04T17:45:25.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.777+0000 7f6739d84640 20 client.4348.objecter _get_session s=0x7f6720073530 osd=1 3 2024-05-04T17:45:25.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.777+0000 7f6739d84640 20 client.4348.objecter put_session s=0x7f6720073530 osd=1 3 2024-05-04T17:45:25.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.777+0000 7f6739d84640 20 client.4348.objecter _get_session s=0x7f6720073530 osd=1 3 2024-05-04T17:45:25.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.777+0000 7f6739d84640 15 client.4348.objecter _session_command_op_remove -1 2 2024-05-04T17:45:25.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.777+0000 7f6739d84640 20 client.4348.objecter get_session s=0x7f6720073530 osd=1 3 2024-05-04T17:45:25.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.777+0000 7f6739d84640 15 client.4348.objecter _session_command_op_assign 1 2 2024-05-04T17:45:25.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.777+0000 7f6739d84640 20 client.4348.objecter put_session s=0x7f6720073530 osd=1 4 2024-05-04T17:45:25.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.777+0000 7f6739d84640 10 client.4348.objecter _send_command 2 2024-05-04T17:45:25.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.777+0000 7f6739d84640 1 -- 172.21.15.19:0/908760950 --> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f673406a700 con 0x7f6720073c00 2024-05-04T17:45:25.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.788+0000 7f21baffd640 1 -- 172.21.15.19:0/1644027812 <== osd.0 v2:172.21.15.19:6808/2236557705 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f21c40b2370 con 0x7f21a8073c00 2024-05-04T17:45:25.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.788+0000 7f21baffd640 10 client.4351.objecter ms_dispatch 0x7f21c4000b90 command_reply(tid 2: 0 ) 2024-05-04T17:45:25.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.788+0000 7f21baffd640 10 client.4351.objecter _finish_command 2 = system:0 2024-05-04T17:45:25.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.788+0000 7f21baffd640 20 client.4351.objecter put_session s=0x7f21a8073530 osd=0 4 2024-05-04T17:45:25.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.788+0000 7f21baffd640 15 client.4351.objecter _session_command_op_remove 0 2 2024-05-04T17:45:25.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 10 librados: watch_flush enter 2024-05-04T17:45:25.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 10 librados: watch_flush exit 2024-05-04T17:45:25.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 10 client.4351.objecter close_session for osd.0 2024-05-04T17:45:25.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 1 -- 172.21.15.19:0/1644027812 >> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] conn(0x7f21a8073c00 msgr2=0x7f21a8076020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] conn(0x7f21a8073c00 0x7f21a8076020 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 20 client.4351.objecter put_session s=0x7f21a8073530 osd=0 1 2024-05-04T17:45:25.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 20 client.4351.objecter shutdown clearing up homeless session... 2024-05-04T17:45:25.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 10 client.4351.objecter successfully canceled tick 2024-05-04T17:45:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 1 -- 172.21.15.19:0/1644027812 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f21a806d7c0 msgr2=0x7f21a806fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f21a806d7c0 0x7f21a806fc80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f21c41350b0 tx=0x7f21b4007a40 comp rx=0 tx=0).stop 2024-05-04T17:45:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 10 monclient: shutdown 2024-05-04T17:45:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 1 -- 172.21.15.19:0/1644027812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21c40a9720 msgr2=0x7f21c4140c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.789+0000 7f21cc570640 1 --2- 172.21.15.19:0/1644027812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21c40a9720 0x7f21c4140c00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f21c000c470 tx=0x7f21c000c930 comp rx=0 tx=0).stop 2024-05-04T17:45:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.790+0000 7f21cab0d640 1 -- 172.21.15.19:0/1644027812 reap_dead start 2024-05-04T17:45:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.790+0000 7f21cc570640 1 -- 172.21.15.19:0/1644027812 shutdown_connections 2024-05-04T17:45:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.790+0000 7f21cc570640 1 -- 172.21.15.19:0/1644027812 >> 172.21.15.19:0/1644027812 conn(0x7f21c4067780 msgr2=0x7f21c40b3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.790+0000 7f21cc570640 1 -- 172.21.15.19:0/1644027812 shutdown_connections 2024-05-04T17:45:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.790+0000 7f21cc570640 1 -- 172.21.15.19:0/1644027812 wait complete. 2024-05-04T17:45:25.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.790+0000 7f21cc570640 1 librados: shutdown 2024-05-04T17:45:25.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.806+0000 7f775affd640 1 -- 172.21.15.19:0/3138761156 <== osd.7 v2:172.21.15.186:6826/1403900459 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7764011e50 con 0x7f774c073c00 2024-05-04T17:45:25.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.806+0000 7f775affd640 10 client.4271.objecter ms_dispatch 0x7f776c000b90 command_reply(tid 1: 0 ) 2024-05-04T17:45:25.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.806+0000 7f775affd640 10 client.4271.objecter _finish_command 1 = system:0 2024-05-04T17:45:25.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.806+0000 7f775affd640 20 client.4271.objecter put_session s=0x7f774c073530 osd=7 4 2024-05-04T17:45:25.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.806+0000 7f775affd640 15 client.4271.objecter _session_command_op_remove 7 1 2024-05-04T17:45:25.808 INFO:teuthology.orchestra.run.smithi019.stdout:85899345929 2024-05-04T17:45:25.808 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-04T17:45:25.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.812+0000 7f671bfff640 1 -- 172.21.15.19:0/908760950 <== osd.1 v2:172.21.15.19:6800/3360656308 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f673406a700 con 0x7f6720073c00 2024-05-04T17:45:25.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.812+0000 7f671bfff640 10 client.4348.objecter ms_dispatch 0x7f6734000b90 command_reply(tid 2: 0 ) 2024-05-04T17:45:25.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.812+0000 7f671bfff640 10 client.4348.objecter _finish_command 2 = system:0 2024-05-04T17:45:25.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.812+0000 7f671bfff640 20 client.4348.objecter put_session s=0x7f6720073530 osd=1 4 2024-05-04T17:45:25.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.812+0000 7f671bfff640 15 client.4348.objecter _session_command_op_remove 1 2 2024-05-04T17:45:25.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 10 librados: watch_flush enter 2024-05-04T17:45:25.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 10 librados: watch_flush exit 2024-05-04T17:45:25.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 10 client.4348.objecter close_session for osd.1 2024-05-04T17:45:25.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 1 -- 172.21.15.19:0/908760950 >> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] conn(0x7f6720073c00 msgr2=0x7f6720076020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] conn(0x7f6720073c00 0x7f6720076020 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 20 client.4348.objecter put_session s=0x7f6720073530 osd=1 1 2024-05-04T17:45:25.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 20 client.4348.objecter shutdown clearing up homeless session... 2024-05-04T17:45:25.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 10 client.4348.objecter successfully canceled tick 2024-05-04T17:45:25.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 1 -- 172.21.15.19:0/908760950 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f672006d7c0 msgr2=0x7f672006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f672006d7c0 0x7f672006fc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6724009710 tx=0x7f6724006c90 comp rx=0 tx=0).stop 2024-05-04T17:45:25.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 10 monclient: shutdown 2024-05-04T17:45:25.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 1 -- 172.21.15.19:0/908760950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67341423e0 msgr2=0x7f67341d70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6739d84640 1 --2- 172.21.15.19:0/908760950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67341423e0 0x7f67341d70c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f672c00b990 tx=0x7f672c00be50 comp rx=0 tx=0).stop 2024-05-04T17:45:25.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.813+0000 7f6733fff640 1 -- 172.21.15.19:0/908760950 reap_dead start 2024-05-04T17:45:25.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.814+0000 7f6739d84640 1 -- 172.21.15.19:0/908760950 shutdown_connections 2024-05-04T17:45:25.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.814+0000 7f6739d84640 1 -- 172.21.15.19:0/908760950 >> 172.21.15.19:0/908760950 conn(0x7f67340b25e0 msgr2=0x7f6734137530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.815+0000 7f6739d84640 1 -- 172.21.15.19:0/908760950 shutdown_connections 2024-05-04T17:45:25.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.815+0000 7f6739d84640 1 -- 172.21.15.19:0/908760950 wait complete. 2024-05-04T17:45:25.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.815+0000 7f6739d84640 1 librados: shutdown 2024-05-04T17:45:25.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.815+0000 7f5a4dad0640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.815+0000 7f5a4dad0640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.815+0000 7f5a4dad0640 10 monclient: monmap: 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.816049+0000 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.816049+0000 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.818 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.819 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.816+0000 7f5a4dad0640 1 Processor -- start 2024-05-04T17:45:25.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.816+0000 7f5a4dad0640 1 -- start start 2024-05-04T17:45:25.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.816+0000 7f5a4dad0640 10 monclient: init 2024-05-04T17:45:25.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-05-04T17:45:25.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5a40039020 0x7f5a400c9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 10 monclient(hunting): picked mon.noname-c con 0x7f5a40039020 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:25.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 0x7f5a400cc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 10 monclient(hunting): picked mon.noname-b con 0x7f5a40035680 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a400cc640 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:25.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a40037420 con 0x7f5a400cc640 2024-05-04T17:45:25.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f5a40039400 con 0x7f5a40039020 2024-05-04T17:45:25.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f5a40035c30 con 0x7f5a40035680 2024-05-04T17:45:25.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.817+0000 7f5a4dad0640 10 monclient(hunting): authenticate will time out at 1347.931315s 2024-05-04T17:45:25.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.819+0000 7f5a47fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 0x7f5a400cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.819+0000 7f5a47fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 0x7f5a400cc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:37030/0 (socket says 172.21.15.19:37030) 2024-05-04T17:45:25.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.819+0000 7f5a47fff640 1 -- 172.21.15.19:0/1714976587 learned_addr learned my addr 172.21.15.19:0/1714976587 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.819+0000 7f5a47fff640 10 monclient(hunting): get_auth_request con 0x7f5a40035680 auth_method 0 2024-05-04T17:45:25.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.819+0000 7f5a47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.819+0000 7f5a47fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.819+0000 7f5a47fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.820+0000 7f5a47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.820+0000 7f5a47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.820+0000 7f5a47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.822+0000 7f5a4cace640 1 --2- 172.21.15.19:0/1714976587 >> v2:172.21.15.186:3300/0 conn(0x7f5a40039020 0x7f5a400c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.823+0000 7f5a4cace640 10 monclient(hunting): get_auth_request con 0x7f5a40039020 auth_method 0 2024-05-04T17:45:25.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.823+0000 7f5a4cace640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.823+0000 7f5a4cace640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.823+0000 7f5a4cace640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.824+0000 7f5a47fff640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-05-04T17:45:25.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.824+0000 7f5a47fff640 1 -- 172.21.15.19:0/1714976587 >> v2:172.21.15.186:3300/0 conn(0x7f5a40039020 msgr2=0x7f5a400c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.824+0000 7f5a47fff640 1 --2- 172.21.15.19:0/1714976587 >> v2:172.21.15.186:3300/0 conn(0x7f5a40039020 0x7f5a400c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.824+0000 7f5a47fff640 1 -- 172.21.15.19:0/1714976587 >> v1:172.21.15.186:6789/0 conn(0x7f5a400cc640 legacy=0x7f5a40034460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.825+0000 7f5a47fff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.825+0000 7f5a47fff640 1 monclient: found mon.noname-b 2024-05-04T17:45:25.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.825+0000 7f5a47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.825+0000 7f5a47fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.825+0000 7f5a47fff640 1 -- 172.21.15.19:0/1714976587 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a40037690 con 0x7f5a40035680 2024-05-04T17:45:25.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.825+0000 7f5a4cace640 1 --2- 172.21.15.19:0/1714976587 >> v2:172.21.15.186:3300/0 conn(0x7f5a40039020 0x7f5a400c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-04T17:45:25.834 INFO:teuthology.orchestra.run.smithi019.stdout:90194313225 2024-05-04T17:45:25.835 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-04T17:45:25.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.830+0000 7f5a47fff640 1 --2- 172.21.15.19:0/1714976587 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 0x7f5a400cc100 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a34009870 tx=0x7f5a3402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edf9e09028285fdd server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 1 -- 172.21.15.19:0/1714976587 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a34035020 con 0x7f5a40035680 2024-05-04T17:45:25.837 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 10 monclient: dump: 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.838 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 1 -- 172.21.15.19:0/1714976587 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a34031640 con 0x7f5a40035680 2024-05-04T17:45:25.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 1 -- 172.21.15.19:0/1714976587 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3402d040 con 0x7f5a40035680 2024-05-04T17:45:25.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:25.839 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a477fe640 10 monclient: dump: 2024-05-04T17:45:25.839 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.839 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.839 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.840 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.840 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.840 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.840 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.840 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.840 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.840 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a4dad0640 5 monclient: authenticate success, global_id 4277 2024-05-04T17:45:25.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a4dad0640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:25.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.836+0000 7f5a4dad0640 4 set_mon_vals no callback set 2024-05-04T17:45:25.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.837+0000 7f5a4dad0640 10 monclient: shutdown 2024-05-04T17:45:25.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.837+0000 7f5a4dad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.837+0000 7f5a4dad0640 1 -- 172.21.15.19:0/1714976587 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 msgr2=0x7f5a400cc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.837+0000 7f5a4dad0640 1 --2- 172.21.15.19:0/1714976587 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 0x7f5a400cc100 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5a34009870 tx=0x7f5a3402b8f0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.838+0000 7f5a4dad0640 1 -- 172.21.15.19:0/1714976587 shutdown_connections 2024-05-04T17:45:25.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.838+0000 7f5a4dad0640 1 --2- 172.21.15.19:0/1714976587 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 0x7f5a400cc100 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.838+0000 7f5a4dad0640 1 --2- 172.21.15.19:0/1714976587 >> v2:172.21.15.186:3300/0 conn(0x7f5a40039020 0x7f5a400c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.838+0000 7f5a4dad0640 1 -- 172.21.15.19:0/1714976587 >> 172.21.15.19:0/1714976587 conn(0x7f5a40006610 msgr2=0x7f5a40006a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.838+0000 7f5a4dad0640 1 -- 172.21.15.19:0/1714976587 shutdown_connections 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.838+0000 7f5a4dad0640 1 -- 172.21.15.19:0/1714976587 wait complete. 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f5a4dad0640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f5a4dad0640 10 monclient: monmap: 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:25.843 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f776b59e640 10 client.4271.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-04T17:45:25.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f776b59e640 15 client.4271.objecter _session_command_op_assign -1 2 2024-05-04T17:45:25.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f776b59e640 20 client.4271.objecter _get_session s=0x7f774c073530 osd=7 3 2024-05-04T17:45:25.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f776b59e640 20 client.4271.objecter put_session s=0x7f774c073530 osd=7 3 2024-05-04T17:45:25.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f776b59e640 20 client.4271.objecter _get_session s=0x7f774c073530 osd=7 3 2024-05-04T17:45:25.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f776b59e640 15 client.4271.objecter _session_command_op_remove -1 2 2024-05-04T17:45:25.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f776b59e640 20 client.4271.objecter get_session s=0x7f774c073530 osd=7 3 2024-05-04T17:45:25.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f776b59e640 15 client.4271.objecter _session_command_op_assign 7 2 2024-05-04T17:45:25.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f776b59e640 20 client.4271.objecter put_session s=0x7f774c073530 osd=7 4 2024-05-04T17:45:25.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f776b59e640 10 client.4271.objecter _send_command 2 2024-05-04T17:45:25.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.839+0000 7f776b59e640 1 -- 172.21.15.19:0/3138761156 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f776c150890 con 0x7f774c073c00 2024-05-04T17:45:25.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.840+0000 7f5a4dad0640 1 librados: starting msgr at 2024-05-04T17:45:25.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.840+0000 7f5a4dad0640 1 librados: starting objecter 2024-05-04T17:45:25.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.840+0000 7f5a4dad0640 1 Processor -- start 2024-05-04T17:45:25.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.840+0000 7f5a4dad0640 1 -- start start 2024-05-04T17:45:25.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.840+0000 7f5a4dad0640 1 librados: setting wanted keys 2024-05-04T17:45:25.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.840+0000 7f5a4dad0640 1 librados: calling monclient init 2024-05-04T17:45:25.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.840+0000 7f5a4dad0640 10 monclient: init 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 0x7f5a4002fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 10 monclient(hunting): picked mon.noname-c con 0x7f5a40035680 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a40039020 0x7f5a40033350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 10 monclient(hunting): picked mon.noname-a con 0x7f5a40039020 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a400cc640 0x7f5a40033890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 10 monclient(hunting): picked mon.noname-b con 0x7f5a400cc640 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5a400c9d40 con 0x7f5a40039020 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5a400c9bc0 con 0x7f5a400cc640 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f5a400ccad0 con 0x7f5a40035680 2024-05-04T17:45:25.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.841+0000 7f5a4dad0640 10 monclient(hunting): authenticate will time out at 1347.955226s 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.842+0000 7f5a4d2cf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a400cc640 0x7f5a40033890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.842+0000 7f5a47fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a40039020 0x7f5a40033350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.842+0000 7f5a4d2cf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a400cc640 0x7f5a40033890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36348/0 (socket says 172.21.15.19:36348) 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.842+0000 7f5a4d2cf640 1 -- 172.21.15.19:0/2669308439 learned_addr learned my addr 172.21.15.19:0/2669308439 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.842+0000 7f5a4d2cf640 10 monclient(hunting): get_auth_request con 0x7f5a400cc640 auth_method 0 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.842+0000 7f5a4d2cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.842+0000 7f5a4d2cf640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.842+0000 7f5a4d2cf640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.843+0000 7f5a4cace640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 0x7f5a4002fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.843+0000 7f5a4cace640 10 monclient(hunting): get_auth_request con 0x7f5a40035680 auth_method 0 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.843+0000 7f5a4cace640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.843+0000 7f5a4cace640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.843+0000 7f5a4cace640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.843+0000 7f5a4d2cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.843+0000 7f5a4d2cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.843+0000 7f5a4d2cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.843+0000 7f5a47fff640 10 monclient(hunting): get_auth_request con 0x7f5a40039020 auth_method 0 2024-05-04T17:45:25.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.843+0000 7f5a47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.843+0000 7f5a47fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.843+0000 7f5a47fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.844+0000 7f5a4cace640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.844+0000 7f5a4cace640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.844+0000 7f5a4cace640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.847+0000 7f775affd640 1 -- 172.21.15.19:0/3138761156 <== osd.7 v2:172.21.15.186:6826/1403900459 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7760004040 con 0x7f774c073c00 2024-05-04T17:45:25.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.847+0000 7f775affd640 10 client.4271.objecter ms_dispatch 0x7f776c000b90 command_reply(tid 2: 0 ) 2024-05-04T17:45:25.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.847+0000 7f775affd640 10 client.4271.objecter _finish_command 2 = system:0 2024-05-04T17:45:25.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.847+0000 7f775affd640 20 client.4271.objecter put_session s=0x7f774c073530 osd=7 4 2024-05-04T17:45:25.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.847+0000 7f775affd640 15 client.4271.objecter _session_command_op_remove 7 2 2024-05-04T17:45:25.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.845+0000 7f5a47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.845+0000 7f5a47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.845+0000 7f5a47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.850+0000 7f5a4cace640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-05-04T17:45:25.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.850+0000 7f5a4cace640 1 -- 172.21.15.19:0/2669308439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a400cc640 msgr2=0x7f5a40033890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.850+0000 7f5a4cace640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a400cc640 0x7f5a40033890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.850+0000 7f5a4cace640 1 -- 172.21.15.19:0/2669308439 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a40039020 msgr2=0x7f5a40033350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.850+0000 7f5a4cace640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a40039020 0x7f5a40033350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.850+0000 7f5a4cace640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.850+0000 7f5a4cace640 1 monclient: found mon.noname-c 2024-05-04T17:45:25.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.850+0000 7f5a4cace640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.850+0000 7f5a4cace640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.850+0000 7f5a4cace640 1 -- 172.21.15.19:0/2669308439 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a40032970 con 0x7f5a40035680 2024-05-04T17:45:25.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.853+0000 7f93daffd640 1 -- 172.21.15.19:0/1360227486 <== osd.2 v2:172.21.15.19:6816/3744614279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f93ec0aeec0 con 0x7f93cc073bb0 2024-05-04T17:45:25.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.853+0000 7f93daffd640 10 client.4274.objecter ms_dispatch 0x7f93ec000b90 command_reply(tid 1: 0 ) 2024-05-04T17:45:25.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.853+0000 7f93daffd640 10 client.4274.objecter _finish_command 1 = system:0 2024-05-04T17:45:25.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.853+0000 7f93daffd640 20 client.4274.objecter put_session s=0x7f93cc0734e0 osd=2 4 2024-05-04T17:45:25.854 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.853+0000 7f93daffd640 15 client.4274.objecter _session_command_op_remove 2 1 2024-05-04T17:45:25.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.853+0000 7f776b59e640 10 librados: watch_flush enter 2024-05-04T17:45:25.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 10 librados: watch_flush exit 2024-05-04T17:45:25.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 10 client.4271.objecter close_session for osd.7 2024-05-04T17:45:25.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 1 -- 172.21.15.19:0/3138761156 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x7f774c073c00 msgr2=0x7f774c076020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 1 --2- 172.21.15.19:0/3138761156 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x7f774c073c00 0x7f774c076020 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 20 client.4271.objecter put_session s=0x7f774c073530 osd=7 1 2024-05-04T17:45:25.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 20 client.4271.objecter shutdown clearing up homeless session... 2024-05-04T17:45:25.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 10 client.4271.objecter successfully canceled tick 2024-05-04T17:45:25.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 1 -- 172.21.15.19:0/3138761156 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f774c06d7c0 msgr2=0x7f774c06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 1 --2- 172.21.15.19:0/3138761156 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f774c06d7c0 0x7f774c06fc80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f775c0047c0 tx=0x7f775c009120 comp rx=0 tx=0).stop 2024-05-04T17:45:25.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 10 monclient: shutdown 2024-05-04T17:45:25.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 1 -- 172.21.15.19:0/3138761156 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f776c153790 msgr2=0x7f776c14e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776b59e640 1 --2- 172.21.15.19:0/3138761156 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f776c153790 0x7f776c14e5e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f776400c7e0 tx=0x7f776400cca0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.856 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.854+0000 7f776ad9d640 1 -- 172.21.15.19:0/3138761156 reap_dead start 2024-05-04T17:45:25.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.857+0000 7f5a4cace640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 0x7f5a4002fe80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00e7e0 tx=0x7f5a3c00eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.858+0000 7f776b59e640 1 -- 172.21.15.19:0/3138761156 shutdown_connections 2024-05-04T17:45:25.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.858+0000 7f776b59e640 1 -- 172.21.15.19:0/3138761156 >> 172.21.15.19:0/3138761156 conn(0x7f776c0b1120 msgr2=0x7f776c0b2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.858+0000 7f5a457fa640 10 client.?.objecter ms_handle_connect 0x7f5a40035680 2024-05-04T17:45:25.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.858+0000 7f5a457fa640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:25.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.858+0000 7f776b59e640 1 -- 172.21.15.19:0/3138761156 shutdown_connections 2024-05-04T17:45:25.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.858+0000 7f776b59e640 1 -- 172.21.15.19:0/3138761156 wait complete. 2024-05-04T17:45:25.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.859+0000 7f776b59e640 1 librados: shutdown 2024-05-04T17:45:25.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 1 -- 172.21.15.19:0/2669308439 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c00f040 con 0x7f5a40035680 2024-05-04T17:45:25.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:25.866 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 10 monclient: dump: 2024-05-04T17:45:25.866 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.866 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.866 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.866 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 1 -- 172.21.15.19:0/2669308439 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a3c004030 con 0x7f5a40035680 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 1 -- 172.21.15.19:0/2669308439 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0043e0 con 0x7f5a40035680 2024-05-04T17:45:25.867 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a457fa640 10 monclient: dump: 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.868 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a467fc640 4 set_mon_vals no callback set 2024-05-04T17:45:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a4dad0640 5 monclient: authenticate success, global_id 4280 2024-05-04T17:45:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a4dad0640 10 monclient: _renew_subs 2024-05-04T17:45:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a4dad0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a4dad0640 1 -- 172.21.15.19:0/2669308439 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a40032cb0 con 0x7f5a40035680 2024-05-04T17:45:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a4dad0640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a4dad0640 10 monclient: _renew_subs 2024-05-04T17:45:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a4dad0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a4dad0640 1 -- 172.21.15.19:0/2669308439 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a400303c0 con 0x7f5a40035680 2024-05-04T17:45:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.865+0000 7f5a4dad0640 1 librados: init done 2024-05-04T17:45:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.867+0000 7f5a4dad0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.867+0000 7f5a4dad0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.870 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.868+0000 7f5a4dad0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:25.871 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.868+0000 7f5a4dad0640 1 -- 172.21.15.19:0/2669308439 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a40008630 con 0x7f5a40035680 2024-05-04T17:45:25.878 INFO:teuthology.orchestra.run.smithi019.stdout:60129542150 2024-05-04T17:45:25.878 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-04T17:45:25.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.882+0000 7f93eb59e640 10 client.4274.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-05-04T17:45:25.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.882+0000 7f93eb59e640 15 client.4274.objecter _session_command_op_assign -1 2 2024-05-04T17:45:25.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.882+0000 7f93eb59e640 20 client.4274.objecter _get_session s=0x7f93cc0734e0 osd=2 3 2024-05-04T17:45:25.883 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.882+0000 7f93eb59e640 20 client.4274.objecter put_session s=0x7f93cc0734e0 osd=2 3 2024-05-04T17:45:25.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.882+0000 7f93eb59e640 20 client.4274.objecter _get_session s=0x7f93cc0734e0 osd=2 3 2024-05-04T17:45:25.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.882+0000 7f93eb59e640 15 client.4274.objecter _session_command_op_remove -1 2 2024-05-04T17:45:25.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.882+0000 7f93eb59e640 20 client.4274.objecter get_session s=0x7f93cc0734e0 osd=2 3 2024-05-04T17:45:25.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.882+0000 7f93eb59e640 15 client.4274.objecter _session_command_op_assign 2 2 2024-05-04T17:45:25.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.882+0000 7f93eb59e640 20 client.4274.objecter put_session s=0x7f93cc0734e0 osd=2 4 2024-05-04T17:45:25.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.882+0000 7f93eb59e640 10 client.4274.objecter _send_command 2 2024-05-04T17:45:25.884 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.882+0000 7f93eb59e640 1 -- 172.21.15.19:0/1360227486 --> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f93e40cc750 con 0x7f93cc073bb0 2024-05-04T17:45:25.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.885+0000 7fd3b2f15640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.885+0000 7fd3b2f15640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.885+0000 7fd3b2f15640 10 monclient: monmap: 2024-05-04T17:45:25.888 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.888 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.888 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.885916+0000 2024-05-04T17:45:25.888 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.885916+0000 2024-05-04T17:45:25.888 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.888 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.888 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.888 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.888 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.889 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.889 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.889 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.887+0000 7fd3b2f15640 1 Processor -- start 2024-05-04T17:45:25.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.887+0000 7fd3b2f15640 1 -- start start 2024-05-04T17:45:25.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.887+0000 7fd3b2f15640 10 monclient: init 2024-05-04T17:45:25.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-05-04T17:45:25.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd3ac0aa140 0x7fd3ac0aa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 10 monclient(hunting): picked mon.noname-b con 0x7fd3ac0aa140 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fd3ac0acb00 0x7fd3ac1441a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 10 monclient(hunting): picked mon.noname-a con 0x7fd3ac0acb00 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:25.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd3ac0ab230 0x7fd3ac147430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 10 monclient(hunting): picked mon.noname-c con 0x7fd3ac0ab230 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:25.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fd3ac0ac670 con 0x7fd3ac0acb00 2024-05-04T17:45:25.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.890 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd3ac146f10 con 0x7fd3ac0ab230 2024-05-04T17:45:25.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fd3ac1446e0 con 0x7fd3ac0aa140 2024-05-04T17:45:25.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.888+0000 7fd3b2f15640 10 monclient(hunting): authenticate will time out at 1348.002525s 2024-05-04T17:45:25.891 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.889+0000 7fca01c5c640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.889+0000 7fca01c5c640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.889+0000 7fca01c5c640 10 monclient: monmap: 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.890164+0000 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.890164+0000 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.892 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.889+0000 7fd3b14b2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd3ac0ab230 0x7fd3ac147430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7fca01c5c640 1 Processor -- start 2024-05-04T17:45:25.893 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7f93daffd640 1 -- 172.21.15.19:0/1360227486 <== osd.2 v2:172.21.15.19:6816/3744614279 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f93e40cc750 con 0x7f93cc073bb0 2024-05-04T17:45:25.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7f93daffd640 10 client.4274.objecter ms_dispatch 0x7f93ec000b90 command_reply(tid 2: 0 ) 2024-05-04T17:45:25.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7f93daffd640 10 client.4274.objecter _finish_command 2 = system:0 2024-05-04T17:45:25.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7f93daffd640 20 client.4274.objecter put_session s=0x7f93cc0734e0 osd=2 4 2024-05-04T17:45:25.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7f93daffd640 15 client.4274.objecter _session_command_op_remove 2 2 2024-05-04T17:45:25.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7fca01c5c640 1 -- start start 2024-05-04T17:45:25.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7fd3b14b2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd3ac0ab230 0x7fd3ac147430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36354/0 (socket says 172.21.15.19:36354) 2024-05-04T17:45:25.894 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7fd3b14b2640 1 -- 172.21.15.19:0/2616110697 learned_addr learned my addr 172.21.15.19:0/2616110697 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7fd3b14b2640 10 monclient(hunting): get_auth_request con 0x7fd3ac0ab230 auth_method 0 2024-05-04T17:45:25.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7fd3b14b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7fd3b14b2640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7fd3b14b2640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.895 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.890+0000 7fca01c5c640 10 monclient: init 2024-05-04T17:45:25.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fd3abfff640 1 --2- 172.21.15.19:0/2616110697 >> v2:172.21.15.19:3300/0 conn(0x7fd3ac0acb00 0x7fd3ac1441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fd3abfff640 10 monclient(hunting): get_auth_request con 0x7fd3ac0acb00 auth_method 0 2024-05-04T17:45:25.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fd3abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fd3abfff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fd3abfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7f93eb59e640 10 librados: watch_flush enter 2024-05-04T17:45:25.896 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7f93eb59e640 10 librados: watch_flush exit 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7f93eb59e640 10 client.4274.objecter close_session for osd.2 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7f93eb59e640 1 -- 172.21.15.19:0/1360227486 >> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] conn(0x7f93cc073bb0 msgr2=0x7f93cc075fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7f93eb59e640 1 --2- 172.21.15.19:0/1360227486 >> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] conn(0x7f93cc073bb0 0x7f93cc075fd0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7f93eb59e640 20 client.4274.objecter put_session s=0x7f93cc0734e0 osd=2 1 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7f93eb59e640 20 client.4274.objecter shutdown clearing up homeless session... 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7f93eb59e640 10 client.4274.objecter successfully canceled tick 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.892+0000 7f93eb59e640 1 -- 172.21.15.19:0/1360227486 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f93cc06d770 msgr2=0x7f93cc06fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.892+0000 7f93eb59e640 1 --2- 172.21.15.19:0/1360227486 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f93cc06d770 0x7f93cc06fc30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f93dc001560 tx=0x7f93dc007990 comp rx=0 tx=0).stop 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.892+0000 7f93eb59e640 10 monclient: shutdown 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.892+0000 7f93eb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.892+0000 7f93eb59e640 1 -- 172.21.15.19:0/1360227486 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40c3340 msgr2=0x7f93e40cb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.892+0000 7f93eb59e640 1 --2- 172.21.15.19:0/1360227486 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f93e40c3340 0x7f93e40cb480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f93ec075b10 tx=0x7f93ec069180 comp rx=0 tx=0).stop 2024-05-04T17:45:25.897 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.892+0000 7f93ead9d640 1 -- 172.21.15.19:0/1360227486 reap_dead start 2024-05-04T17:45:25.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fca01c5c640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fca01c5c640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-05-04T17:45:25.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fca01c5c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc9fc0a9720 0x7fc9fc0aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fca01c5c640 10 monclient(hunting): picked mon.noname-b con 0x7fc9fc0a9720 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fca01c5c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc9fc0a9b00 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:25.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fca01c5c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc9fc0b2e90 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:25.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fca01c5c640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9fc0abc40 con 0x7fc9fc0b2e90 2024-05-04T17:45:25.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fca01c5c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9fc146fa0 con 0x7fc9fc0a9b00 2024-05-04T17:45:25.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fca01c5c640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fca01c5c640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc9fc146090 con 0x7fc9fc0a9720 2024-05-04T17:45:25.898 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.892+0000 7fca01c5c640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.892+0000 7fca01c5c640 10 monclient(hunting): authenticate will time out at 1348.005897s 2024-05-04T17:45:25.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fd3b14b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fd3b14b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.891+0000 7fd3b14b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.899 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7fc9fbfff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fc9fc0b2e90 0x7fc9fc13fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:42312/0 (socket says 172.21.15.19:42312) 2024-05-04T17:45:25.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7fc9fbfff640 1 -- 172.21.15.19:0/1391782663 learned_addr learned my addr 172.21.15.19:0/1391782663 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:25.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7f93eb59e640 1 -- 172.21.15.19:0/1360227486 shutdown_connections 2024-05-04T17:45:25.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7f93eb59e640 1 -- 172.21.15.19:0/1360227486 >> 172.21.15.19:0/1360227486 conn(0x7f93e40063d0 msgr2=0x7f93e4036300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7fd3b14b2640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2024-05-04T17:45:25.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7fd3b14b2640 1 -- 172.21.15.19:0/2616110697 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd3ac0aa140 msgr2=0x7fd3ac0aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7fd3b14b2640 1 --2- 172.21.15.19:0/2616110697 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd3ac0aa140 0x7fd3ac0aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7fd3b14b2640 1 -- 172.21.15.19:0/2616110697 >> v2:172.21.15.19:3300/0 conn(0x7fd3ac0acb00 msgr2=0x7fd3ac1441a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7fd3b14b2640 1 --2- 172.21.15.19:0/2616110697 >> v2:172.21.15.19:3300/0 conn(0x7fd3ac0acb00 0x7fd3ac1441a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7fd3b14b2640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.900 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7fd3b14b2640 1 monclient: found mon.noname-c 2024-05-04T17:45:25.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7fd3b14b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7fd3b14b2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.893+0000 7fd3b14b2640 1 -- 172.21.15.19:0/2616110697 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac144890 con 0x7fd3ac0ab230 2024-05-04T17:45:25.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.894+0000 7f93eb59e640 1 -- 172.21.15.19:0/1360227486 shutdown_connections 2024-05-04T17:45:25.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.895+0000 7f93eb59e640 1 -- 172.21.15.19:0/1360227486 wait complete. 2024-05-04T17:45:25.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.895+0000 7f93eb59e640 1 librados: shutdown 2024-05-04T17:45:25.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.896+0000 7fd3b14b2640 1 --2- 172.21.15.19:0/2616110697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac0ab230 0x7fd3ac147430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac06c5a0 tx=0x7fd39c02b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eb6b9d601eaed13 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.898+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3477613165 0 0) 0x7fc9fc146fa0 con 0x7fc9fc0a9b00 2024-05-04T17:45:25.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.898+0000 7fc9fa7fc640 10 monclient(hunting): _negotiate 2024-05-04T17:45:25.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.898+0000 7fc9fa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.898+0000 7fc9fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.898+0000 7fc9fa7fc640 10 monclient(hunting): my global_id is 4366 2024-05-04T17:45:25.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.898+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9e4003520 con 0x7fc9fc0a9b00 2024-05-04T17:45:25.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.901+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4161116648 0 0) 0x7fc9fc0abc40 con 0x7fc9fc0b2e90 2024-05-04T17:45:25.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.901+0000 7fc9fa7fc640 10 monclient(hunting): _negotiate 2024-05-04T17:45:25.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.901+0000 7fc9fa7fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.901+0000 7fc9fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.901+0000 7fc9fa7fc640 10 monclient(hunting): my global_id is 4299 2024-05-04T17:45:25.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.901+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9fc146fa0 con 0x7fc9fc0b2e90 2024-05-04T17:45:25.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 1 -- 172.21.15.19:0/2616110697 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39c0317f0 con 0x7fd3ac0ab230 2024-05-04T17:45:25.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:25.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 10 monclient: dump: 2024-05-04T17:45:25.903 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.903 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 1 -- 172.21.15.19:0/2616110697 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd39c037070 con 0x7fd3ac0ab230 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 1 -- 172.21.15.19:0/2616110697 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39c0317f0 con 0x7fd3ac0ab230 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.902+0000 7fd3ab7fe640 10 monclient: dump: 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.905 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.903+0000 7f5a457fa640 1 -- 172.21.15.19:0/2669308439 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5a3c00c6d0 con 0x7f5a40035680 2024-05-04T17:45:25.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.904+0000 7f5a457fa640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5a2c06d810 0x7f5a2c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.906 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.904+0000 7fd3b2f15640 5 monclient: authenticate success, global_id 4363 2024-05-04T17:45:25.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7f5a47fff640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5a2c06d810 0x7f5a2c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.905+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2261416313 0 0) 0x7fc9e4003520 con 0x7fc9fc0a9b00 2024-05-04T17:45:25.907 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.905+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc9fc0a9720 msgr2=0x7fc9fc0aa890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.905+0000 7fc9fa7fc640 1 --2- 172.21.15.19:0/1391782663 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc9fc0a9720 0x7fc9fc0aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.905+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 >> v1:172.21.15.19:6789/0 conn(0x7fc9fc0b2e90 legacy=0x7fc9fc13fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.905+0000 7fc9fa7fc640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.905+0000 7fc9fa7fc640 1 monclient: found mon.noname-c-legacy 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.905+0000 7fc9fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.186:6789/0 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9fc146370 con 0x7fc9fc0a9b00 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3626739996 0 0) 0x7fc9ec001670 con 0x7fc9fc0a9b00 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient: dump: 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.908 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] but i'm connected to v1:172.21.15.186:6789/0, reconnecting 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 >> v1:172.21.15.186:6789/0 conn(0x7fc9fc0a9b00 legacy=0x7fc9fc146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 1 --2- 172.21.15.19:0/1391782663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9e4003b40 0x7fc9e4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient(hunting): picked mon.b con 0x7fc9e4003b40 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 1 --2- 172.21.15.19:0/1391782663 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9e4004800 0x7fc9e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient(hunting): picked mon.a con 0x7fc9e4004800 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 1 --2- 172.21.15.19:0/1391782663 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc9fc0a9720 0x7fc9e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient(hunting): picked mon.c con 0x7fc9fc0a9720 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc9fc146090 con 0x7fc9e4004800 2024-05-04T17:45:25.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e4005520 con 0x7fc9e4003b40 2024-05-04T17:45:25.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e400bf00 con 0x7fc9fc0a9720 2024-05-04T17:45:25.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.906+0000 7fc9fa7fc640 10 monclient(hunting): _finish_auth 0 2024-05-04T17:45:25.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.904+0000 7fd3b2f15640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:25.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.904+0000 7fd3b2f15640 4 set_mon_vals no callback set 2024-05-04T17:45:25.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.905+0000 7fd3b2f15640 10 monclient: shutdown 2024-05-04T17:45:25.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.905+0000 7fd3b2f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.905+0000 7fd3b2f15640 1 -- 172.21.15.19:0/2616110697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac0ab230 msgr2=0x7fd3ac147430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.905+0000 7fd3b2f15640 1 --2- 172.21.15.19:0/2616110697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac0ab230 0x7fd3ac147430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac06c5a0 tx=0x7fd39c02b890 comp rx=0 tx=0).stop 2024-05-04T17:45:25.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.907+0000 7fd3b2f15640 1 -- 172.21.15.19:0/2616110697 shutdown_connections 2024-05-04T17:45:25.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.907+0000 7fd3b2f15640 1 --2- 172.21.15.19:0/2616110697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac0ab230 0x7fd3ac147430 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.907+0000 7fd3b2f15640 1 --2- 172.21.15.19:0/2616110697 >> v2:172.21.15.19:3300/0 conn(0x7fd3ac0acb00 0x7fd3ac1441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.907+0000 7fd3b2f15640 1 --2- 172.21.15.19:0/2616110697 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd3ac0aa140 0x7fd3ac0aa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.907+0000 7fd3b2f15640 1 -- 172.21.15.19:0/2616110697 >> 172.21.15.19:0/2616110697 conn(0x7fd3ac0b43a0 msgr2=0x7fd3ac0b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.907+0000 7f5a47fff640 10 monclient: get_auth_request con 0x7f5a2c06d810 auth_method 0 2024-05-04T17:45:25.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.907+0000 7fd3b2f15640 1 -- 172.21.15.19:0/2616110697 shutdown_connections 2024-05-04T17:45:25.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.907+0000 7fd3b2f15640 1 -- 172.21.15.19:0/2616110697 wait complete. 2024-05-04T17:45:25.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.908+0000 7fd3b2f15640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.908+0000 7fd3b2f15640 10 monclient: monmap: 2024-05-04T17:45:25.914 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.914 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:25.914 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:25.914 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:25.914 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.914 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.914 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:25.914 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:25.914 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:25.914 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.909+0000 7fd3b2f15640 1 librados: starting msgr at 2024-05-04T17:45:25.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.909+0000 7fd3b2f15640 1 librados: starting objecter 2024-05-04T17:45:25.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.909+0000 7fd3b2f15640 1 Processor -- start 2024-05-04T17:45:25.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fc9fb7fe640 1 --2- 172.21.15.19:0/1391782663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9e4003b40 0x7fc9e4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fc9fb7fe640 10 monclient(hunting): get_auth_request con 0x7fc9e4003b40 auth_method 0 2024-05-04T17:45:25.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fc9fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fc9fb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fc9fb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-04T17:45:25.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.909+0000 7fd3b2f15640 1 -- start start 2024-05-04T17:45:25.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.909+0000 7fd3b2f15640 1 librados: setting wanted keys 2024-05-04T17:45:25.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.909+0000 7fd3b2f15640 1 librados: calling monclient init 2024-05-04T17:45:25.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.909+0000 7fd3b2f15640 10 monclient: init 2024-05-04T17:45:25.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-04T17:45:25.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3ac0aa140 0x7fd3ac145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 10 monclient(hunting): picked mon.noname-a con 0x7fd3ac0aa140 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac0ab230 0x7fd3ac141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 10 monclient(hunting): picked mon.noname-b con 0x7fd3ac0ab230 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd3ac0acb00 0x7fd3ac1e8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 10 monclient(hunting): picked mon.noname-c con 0x7fd3ac0acb00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac147d00 con 0x7fd3ac0aa140 2024-05-04T17:45:25.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac147b80 con 0x7fd3ac0ab230 2024-05-04T17:45:25.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fd3ac14a2e0 con 0x7fd3ac0acb00 2024-05-04T17:45:25.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.910+0000 7fd3b2f15640 10 monclient(hunting): authenticate will time out at 1348.024522s 2024-05-04T17:45:25.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7f5a457fa640 10 client.4280.objecter ms_handle_connect 0x7f5a2c06d810 2024-05-04T17:45:25.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7f5a47fff640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5a2c06d810 0x7f5a2c06fcd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5a34036000 tx=0x7f5a3403c040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7fc9faffd640 1 --2- 172.21.15.19:0/1391782663 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9e4004800 0x7fc9e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7fc9fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7fc9fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7fc9fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-04T17:45:25.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7f85d1c2c640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7f85d1c2c640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7f85d1c2c640 10 monclient: monmap: 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.911936+0000 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.911936+0000 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.919 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7fc9faffd640 10 monclient(hunting): get_auth_request con 0x7fc9e4004800 auth_method 0 2024-05-04T17:45:25.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7fc9faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7fc9faffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.920 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7fc9faffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-05-04T17:45:25.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.911+0000 7fd3b0cb1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3ac0aa140 0x7fd3ac145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.912+0000 7fd3abfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac0ab230 0x7fd3ac141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.912+0000 7fd3b0cb1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3ac0aa140 0x7fd3ac145fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37318/0 (socket says 172.21.15.19:37318) 2024-05-04T17:45:25.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.912+0000 7fd3b0cb1640 1 -- 172.21.15.19:0/1129755580 learned_addr learned my addr 172.21.15.19:0/1129755580 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.912+0000 7fd3b0cb1640 10 monclient(hunting): get_auth_request con 0x7fd3ac0aa140 auth_method 0 2024-05-04T17:45:25.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.912+0000 7fd3b0cb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.912+0000 7fd3b0cb1640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.912+0000 7fd3b0cb1640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.912+0000 7f85d1c2c640 1 Processor -- start 2024-05-04T17:45:25.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.912+0000 7f85d1c2c640 1 -- start start 2024-05-04T17:45:25.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.912+0000 7f85d1c2c640 10 monclient: init 2024-05-04T17:45:25.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.923 INFO:teuthology.orchestra.run.smithi019.stdout:60129542151 2024-05-04T17:45:25.923 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-04T17:45:25.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-05-04T17:45:25.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f85cc0b5390 0x7f85cc147590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 10 monclient(hunting): picked mon.noname-c con 0x7f85cc0b5390 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:25.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85cc1441d0 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:25.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f85cc072050 0x7f85cc0724a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 10 monclient(hunting): picked mon.noname-a con 0x7f85cc072050 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:25.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85cc071930 con 0x7f85cc1441d0 2024-05-04T17:45:25.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f85cc144d20 con 0x7f85cc072050 2024-05-04T17:45:25.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f85cc0b5ff0 con 0x7f85cc0b5390 2024-05-04T17:45:25.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7f85d1c2c640 10 monclient(hunting): authenticate will time out at 1348.027125s 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7fc9faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7fc9faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.913+0000 7fc9faffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 10 monclient(hunting): handle_auth_done global_id 4366 payload 293 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 1 -- 172.21.15.19:0/1391782663 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc9fc0a9720 msgr2=0x7fc9e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 1 --2- 172.21.15.19:0/1391782663 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc9fc0a9720 0x7fc9e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 1 -- 172.21.15.19:0/1391782663 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9e4004800 msgr2=0x7fc9e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 1 --2- 172.21.15.19:0/1391782663 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9e4004800 0x7fc9e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 1 monclient: found mon.b 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 1 -- 172.21.15.19:0/1391782663 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e4003520 con 0x7fc9e4003b40 2024-05-04T17:45:25.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fc9fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fd3abfff640 10 monclient(hunting): get_auth_request con 0x7fd3ac0ab230 auth_method 0 2024-05-04T17:45:25.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fca01c5c640 5 monclient: authenticate success, global_id 4366 2024-05-04T17:45:25.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fca01c5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:25.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fd3abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fd3abfff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.914+0000 7fd3abfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.915+0000 7f85cb7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f85cc0b5390 0x7f85cc147590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.916+0000 7fd3abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.916+0000 7fd3abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.916+0000 7fd3abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.916+0000 7f85cb7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f85cc0b5390 0x7f85cc147590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36366/0 (socket says 172.21.15.19:36366) 2024-05-04T17:45:25.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.916+0000 7f85cb7fe640 1 -- 172.21.15.19:0/1059388783 learned_addr learned my addr 172.21.15.19:0/1059388783 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.916+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request con 0x7f85cc0b5390 auth_method 0 2024-05-04T17:45:25.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.916+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.916+0000 7f85cb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.916+0000 7f85cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.917+0000 7fc9fb7fe640 1 --2- 172.21.15.19:0/1391782663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9e4003b40 0x7fc9e4003f40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009860 tx=0x7fc9e4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d35e0ebdbabee639 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.919+0000 7fd3b0cb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.919+0000 7fd3b0cb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.919+0000 7fd3b0cb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.919+0000 7fd3abfff640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2024-05-04T17:45:25.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.919+0000 7fd3abfff640 1 -- 172.21.15.19:0/1129755580 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd3ac0acb00 msgr2=0x7fd3ac1e8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.919+0000 7fd3abfff640 1 --2- 172.21.15.19:0/1129755580 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd3ac0acb00 0x7fd3ac1e8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.919+0000 7fd3abfff640 1 -- 172.21.15.19:0/1129755580 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3ac0aa140 msgr2=0x7fd3ac145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.919+0000 7fd3abfff640 1 --2- 172.21.15.19:0/1129755580 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3ac0aa140 0x7fd3ac145fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.920+0000 7fd3abfff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.920+0000 7fd3abfff640 1 monclient: found mon.noname-b 2024-05-04T17:45:25.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.920+0000 7fd3abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.920+0000 7fd3abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.920+0000 7fd3abfff640 1 -- 172.21.15.19:0/1129755580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac149670 con 0x7fd3ac0ab230 2024-05-04T17:45:25.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.922+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.922+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.922+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.924+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_done global_id 4372 payload 274 2024-05-04T17:45:25.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.924+0000 7f85cb7fe640 1 -- 172.21.15.19:0/1059388783 >> v2:172.21.15.19:3300/0 conn(0x7f85cc072050 msgr2=0x7f85cc0724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.924+0000 7f85cb7fe640 1 --2- 172.21.15.19:0/1059388783 >> v2:172.21.15.19:3300/0 conn(0x7f85cc072050 0x7f85cc0724a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.924+0000 7f85cb7fe640 1 -- 172.21.15.19:0/1059388783 >> v1:172.21.15.19:6789/0 conn(0x7f85cc1441d0 legacy=0x7f85cc144610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:25.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.924+0000 7f85cb7fe640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.924+0000 7f85cb7fe640 1 monclient: found mon.noname-c 2024-05-04T17:45:25.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.924+0000 7f85cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.924+0000 7f85cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.924+0000 7f85cb7fe640 1 -- 172.21.15.19:0/1059388783 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85cc147ba0 con 0x7f85cc0b5390 2024-05-04T17:45:25.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.927+0000 7fd3a97fa640 10 client.?.objecter ms_handle_connect 0x7fd3ac0ab230 2024-05-04T17:45:25.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.927+0000 7fd3a97fa640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:25.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.927+0000 7fd3abfff640 1 --2- 172.21.15.19:0/1129755580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac0ab230 0x7fd3ac141c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4007d80 tx=0x7fd3a400e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.930+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0037070 con 0x7fc9e4003b40 2024-05-04T17:45:25.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.930+0000 7fc9fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.930+0000 7fc9fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:25.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.930+0000 7fc9fa7fc640 10 monclient: dump: 2024-05-04T17:45:25.936 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.936 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.936 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.936 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.936 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.936 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.936 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.936 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.936 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.936 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.930+0000 7fc9fa7fc640 1 -- 172.21.15.19:0/1391782663 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f0032620 con 0x7fc9e4003b40 2024-05-04T17:45:25.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.930+0000 7fca01c5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:25.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.930+0000 7fc9fa7fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.930+0000 7f85cb7fe640 1 --2- 172.21.15.19:0/1059388783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85cc0b5390 0x7f85cc147590 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0b2c10 tx=0x7f85bc02b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=180e577d4aeadd1b server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.930+0000 7fca01c5c640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:25.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.930+0000 7fca01c5c640 4 set_mon_vals no callback set 2024-05-04T17:45:25.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.931+0000 7fca01c5c640 10 monclient: shutdown 2024-05-04T17:45:25.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.931+0000 7fca01c5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.931+0000 7fca01c5c640 1 -- 172.21.15.19:0/1391782663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9e4003b40 msgr2=0x7fc9e4003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.931+0000 7fca01c5c640 1 --2- 172.21.15.19:0/1391782663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9e4003b40 0x7fc9e4003f40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009860 tx=0x7fc9e4005830 comp rx=0 tx=0).stop 2024-05-04T17:45:25.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.931+0000 7fc9fbfff640 1 -- 172.21.15.19:0/1391782663 reap_dead start 2024-05-04T17:45:25.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.931+0000 7fca01c5c640 1 -- 172.21.15.19:0/1391782663 shutdown_connections 2024-05-04T17:45:25.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.931+0000 7fca01c5c640 1 -- 172.21.15.19:0/1391782663 >> 172.21.15.19:0/1391782663 conn(0x7fc9fc067780 msgr2=0x7fc9fc0b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.931+0000 7fca01c5c640 1 -- 172.21.15.19:0/1391782663 shutdown_connections 2024-05-04T17:45:25.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.931+0000 7fca01c5c640 1 -- 172.21.15.19:0/1391782663 wait complete. 2024-05-04T17:45:25.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.932+0000 7fca01c5c640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.939 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.932+0000 7fca01c5c640 10 monclient: monmap: 2024-05-04T17:45:25.939 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.939 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:25.939 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:25.939 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fca01c5c640 1 librados: starting msgr at 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fca01c5c640 1 librados: starting objecter 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fca01c5c640 1 Processor -- start 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fca01c5c640 1 -- start start 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fca01c5c640 1 librados: setting wanted keys 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fca01c5c640 1 librados: calling monclient init 2024-05-04T17:45:25.940 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fca01c5c640 10 monclient: init 2024-05-04T17:45:25.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 1 -- 172.21.15.19:0/1129755580 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4014420 con 0x7fd3ac0ab230 2024-05-04T17:45:25.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:25.941 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 10 monclient: dump: 2024-05-04T17:45:25.941 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.942 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.942 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.942 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.942 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.942 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.942 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.942 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.942 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.942 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.942 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 1 -- 172.21.15.19:0/1129755580 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a4019070 con 0x7fd3ac0ab230 2024-05-04T17:45:25.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 1 -- 172.21.15.19:0/1129755580 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4014420 con 0x7fd3ac0ab230 2024-05-04T17:45:25.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.943 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3a97fa640 10 monclient: dump: 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.933+0000 7fd3aa7fc640 4 set_mon_vals no callback set 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fd3b2f15640 5 monclient: authenticate success, global_id 4369 2024-05-04T17:45:25.944 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fd3b2f15640 10 monclient: _renew_subs 2024-05-04T17:45:25.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fd3b2f15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fd3b2f15640 1 -- 172.21.15.19:0/1129755580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ac149de0 con 0x7fd3ac0ab230 2024-05-04T17:45:25.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fd3b2f15640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fd3b2f15640 10 monclient: _renew_subs 2024-05-04T17:45:25.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fd3b2f15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fd3b2f15640 1 -- 172.21.15.19:0/1129755580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ac142ff0 con 0x7fd3ac0ab230 2024-05-04T17:45:25.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fd3b2f15640 1 librados: init done 2024-05-04T17:45:25.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-04T17:45:25.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9e4003b40 0x7fc9fc1db8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 10 monclient(hunting): picked mon.noname-b con 0x7fc9e4003b40 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc9fc141790 0x7fc9fc139550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 10 monclient(hunting): picked mon.noname-c con 0x7fc9fc141790 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9fc1dbe00 0x7fc9fc139a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 10 monclient(hunting): picked mon.noname-a con 0x7fc9fc1dbe00 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc9fc0aaa10 con 0x7fc9fc1dbe00 2024-05-04T17:45:25.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc9fc0aa890 con 0x7fc9e4003b40 2024-05-04T17:45:25.946 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc9fc0aab90 con 0x7fc9fc141790 2024-05-04T17:45:25.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.934+0000 7fca01c5c640 10 monclient(hunting): authenticate will time out at 1348.048042s 2024-05-04T17:45:25.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.935+0000 7fc9fbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9fc1dbe00 0x7fc9fc139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.947 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.935+0000 7fd3b2f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.935+0000 7fd3b2f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.935+0000 7fd3b2f15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:25.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.935+0000 7fd3b2f15640 1 -- 172.21.15.19:0/1129755580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3ac04f660 con 0x7fd3ac0ab230 2024-05-04T17:45:25.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.936+0000 7fc9fbfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9fc1dbe00 0x7fc9fc139a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37332/0 (socket says 172.21.15.19:37332) 2024-05-04T17:45:25.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.936+0000 7fc9fbfff640 1 -- 172.21.15.19:0/2398717187 learned_addr learned my addr 172.21.15.19:0/2398717187 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.936+0000 7fc9fbfff640 10 monclient(hunting): get_auth_request con 0x7fc9fc1dbe00 auth_method 0 2024-05-04T17:45:25.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.936+0000 7fc9fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.936+0000 7fc9fbfff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.936+0000 7fc9fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.937+0000 7fc9fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.937+0000 7fc9fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.937+0000 7fc9fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.940+0000 7f5a457fa640 1 -- 172.21.15.19:0/2669308439 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f5a3c057350 con 0x7f5a40035680 2024-05-04T17:45:25.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.940+0000 7f5a457fa640 10 client.4280.objecter ms_dispatch 0x7f5a48000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:25.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.940+0000 7f5a457fa640 3 client.4280.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:25.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.940+0000 7f5a457fa640 3 client.4280.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:25.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.940+0000 7f5a457fa640 20 client.4280.objecter dump_active .. 0 homeless 2024-05-04T17:45:25.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 1 -- 172.21.15.19:0/1059388783 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc038070 con 0x7f85cc0b5390 2024-05-04T17:45:25.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:25.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 10 monclient: dump: 2024-05-04T17:45:25.951 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.951 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 1 -- 172.21.15.19:0/1059388783 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85bc02f050 con 0x7f85cc0b5390 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85d1c2c640 5 monclient: authenticate success, global_id 4372 2024-05-04T17:45:25.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85d1c2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 1 -- 172.21.15.19:0/1059388783 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc031970 con 0x7f85cc0b5390 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85ca7fc640 10 monclient: dump: 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.953 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.954 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.954 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85d1c2c640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:25.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.943+0000 7f85d1c2c640 4 set_mon_vals no callback set 2024-05-04T17:45:25.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.944+0000 7f85d1c2c640 10 monclient: shutdown 2024-05-04T17:45:25.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.944+0000 7f85d1c2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:25.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.944+0000 7f85d1c2c640 1 -- 172.21.15.19:0/1059388783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85cc0b5390 msgr2=0x7f85cc147590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.944+0000 7f85d1c2c640 1 --2- 172.21.15.19:0/1059388783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85cc0b5390 0x7f85cc147590 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f85cc0b2c10 tx=0x7f85bc02b7f0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7f85d1c2c640 1 -- 172.21.15.19:0/1059388783 shutdown_connections 2024-05-04T17:45:25.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7f85d1c2c640 1 --2- 172.21.15.19:0/1059388783 >> v2:172.21.15.19:3300/0 conn(0x7f85cc072050 0x7f85cc0724a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7f85d1c2c640 1 --2- 172.21.15.19:0/1059388783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85cc0b5390 0x7f85cc147590 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7f85d1c2c640 1 -- 172.21.15.19:0/1059388783 >> 172.21.15.19:0/1059388783 conn(0x7f85cc0b2cb0 msgr2=0x7f85cc0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:25.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7f85d1c2c640 1 -- 172.21.15.19:0/1059388783 shutdown_connections 2024-05-04T17:45:25.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7f85d1c2c640 1 -- 172.21.15.19:0/1059388783 wait complete. 2024-05-04T17:45:25.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7fc9fbfff640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2024-05-04T17:45:25.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7fc9fbfff640 1 -- 172.21.15.19:0/2398717187 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc9fc141790 msgr2=0x7fc9fc139550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7fc9fbfff640 1 --2- 172.21.15.19:0/2398717187 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc9fc141790 0x7fc9fc139550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7fc9fbfff640 1 -- 172.21.15.19:0/2398717187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9e4003b40 msgr2=0x7fc9fc1db8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7fc9fbfff640 1 --2- 172.21.15.19:0/2398717187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9e4003b40 0x7fc9fc1db8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7fc9fbfff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7fc9fbfff640 1 monclient: found mon.noname-a 2024-05-04T17:45:25.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7fc9fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7fc9fbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.945+0000 7fc9fbfff640 1 -- 172.21.15.19:0/2398717187 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9fc1db3a0 con 0x7fc9fc1dbe00 2024-05-04T17:45:25.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.946+0000 7f85d1c2c640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.946+0000 7f85d1c2c640 10 monclient: monmap: 2024-05-04T17:45:25.958 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.958 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:25.958 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:25.958 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:25.958 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.958 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.958 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:25.958 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:25.958 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:25.958 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.946+0000 7f85d1c2c640 1 librados: starting msgr at 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.946+0000 7f85d1c2c640 1 librados: starting objecter 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.946+0000 7f85d1c2c640 1 Processor -- start 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 1 -- start start 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 1 librados: setting wanted keys 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 1 librados: calling monclient init 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 10 monclient: init 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f85cc072050 0x7f85cc1401e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 10 monclient(hunting): picked mon.noname-a con 0x7f85cc072050 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f85cc0b5390 0x7f85cc13ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 10 monclient(hunting): picked mon.noname-c con 0x7f85cc0b5390 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:25.959 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85cc1441d0 0x7f85cc13b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 10 monclient(hunting): picked mon.noname-b con 0x7f85cc1441d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc072b60 con 0x7f85cc072050 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc0729e0 con 0x7f85cc1441d0 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f85cc072ce0 con 0x7f85cc0b5390 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.947+0000 7f85d1c2c640 10 monclient(hunting): authenticate will time out at 1348.061543s 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.948+0000 7f85cb7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f85cc072050 0x7f85cc1401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.949+0000 7f85cb7fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f85cc072050 0x7f85cc1401e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37336/0 (socket says 172.21.15.19:37336) 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.949+0000 7f85cb7fe640 1 -- 172.21.15.19:0/2523051078 learned_addr learned my addr 172.21.15.19:0/2523051078 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:25.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.949+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request con 0x7f85cc072050 auth_method 0 2024-05-04T17:45:25.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.949+0000 7f85cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:25.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.949+0000 7f85cb7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:25.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.949+0000 7f85cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:25.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.950+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:25.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.950+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:25.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.950+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:25.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.950+0000 7fc9fbfff640 1 --2- 172.21.15.19:0/2398717187 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9fc1dbe00 0x7fc9fc139a90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc0b3270 tx=0x7fc9f400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.950+0000 7fc9e3fff640 10 client.?.objecter ms_handle_connect 0x7fc9fc1dbe00 2024-05-04T17:45:25.961 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.950+0000 7fc9e3fff640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.953+0000 7f85cb7fe640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2024-05-04T17:45:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.953+0000 7f85cb7fe640 1 -- 172.21.15.19:0/2523051078 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f85cc0b5390 msgr2=0x7f85cc13ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.953+0000 7f85cb7fe640 1 --2- 172.21.15.19:0/2523051078 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f85cc0b5390 0x7f85cc13ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.953+0000 7f85cb7fe640 1 -- 172.21.15.19:0/2523051078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85cc1441d0 msgr2=0x7f85cc13b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.953+0000 7f85cb7fe640 1 --2- 172.21.15.19:0/2523051078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85cc1441d0 0x7f85cc13b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.954+0000 7f85cb7fe640 10 monclient: _finish_hunting 0 2024-05-04T17:45:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.954+0000 7f85cb7fe640 1 monclient: found mon.noname-a 2024-05-04T17:45:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.954+0000 7f85cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.954+0000 7f85cb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.954+0000 7f85cb7fe640 1 -- 172.21.15.19:0/2523051078 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85cc149ab0 con 0x7f85cc072050 2024-05-04T17:45:25.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.956+0000 7fd3a97fa640 1 -- 172.21.15.19:0/1129755580 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd3a40145c0 con 0x7fd3ac0ab230 2024-05-04T17:45:25.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.956+0000 7fd3a97fa640 1 --2- 172.21.15.19:0/1129755580 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fd39806dbc0 0x7fd398070080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.957+0000 7fd3b0cb1640 1 --2- 172.21.15.19:0/1129755580 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fd39806dbc0 0x7fd398070080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:25.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.957+0000 7fd3b0cb1640 10 monclient: get_auth_request con 0x7fd39806dbc0 auth_method 0 2024-05-04T17:45:25.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.957+0000 7fd3a97fa640 10 client.4369.objecter ms_handle_connect 0x7fd39806dbc0 2024-05-04T17:45:25.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.957+0000 7fd3b0cb1640 1 --2- 172.21.15.19:0/1129755580 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fd39806dbc0 0x7fd398070080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009a40 tx=0x7fd3a00091c0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.959+0000 7f85cb7fe640 1 --2- 172.21.15.19:0/2523051078 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f85cc072050 0x7f85cc1401e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f85bc000c00 tx=0x7f85bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:25.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.959+0000 7f85b3fff640 10 client.?.objecter ms_handle_connect 0x7f85cc072050 2024-05-04T17:45:25.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.959+0000 7f85b3fff640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:25.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.961+0000 7fd3a97fa640 1 -- 172.21.15.19:0/1129755580 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fd3a400edd0 con 0x7fd3ac0ab230 2024-05-04T17:45:25.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.961+0000 7fd3a97fa640 10 client.4369.objecter ms_dispatch 0x7fd3ac000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:25.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.961+0000 7fd3a97fa640 3 client.4369.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:25.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.961+0000 7fd3a97fa640 3 client.4369.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:25.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.961+0000 7fd3a97fa640 20 client.4369.objecter dump_active .. 0 homeless 2024-05-04T17:45:25.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 1 -- 172.21.15.19:0/2398717187 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4017070 con 0x7fc9fc1dbe00 2024-05-04T17:45:25.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:25.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 10 monclient: dump: 2024-05-04T17:45:25.967 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 1 -- 172.21.15.19:0/2398717187 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f400aba0 con 0x7fc9fc1dbe00 2024-05-04T17:45:25.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 1 -- 172.21.15.19:0/2398717187 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f40070d0 con 0x7fc9fc1dbe00 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9e3fff640 10 monclient: dump: 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.969 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.963+0000 7fc9f9ffb640 4 set_mon_vals no callback set 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.964+0000 7fca01c5c640 5 monclient: authenticate success, global_id 4305 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.964+0000 7fca01c5c640 10 monclient: _renew_subs 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.964+0000 7fca01c5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.964+0000 7fca01c5c640 1 -- 172.21.15.19:0/2398717187 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9fc139fd0 con 0x7fc9fc1dbe00 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.964+0000 7fca01c5c640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.964+0000 7fca01c5c640 10 monclient: _renew_subs 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.964+0000 7fca01c5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.964+0000 7fca01c5c640 1 -- 172.21.15.19:0/2398717187 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9fc0ad0e0 con 0x7fc9fc1dbe00 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.964+0000 7fca01c5c640 1 librados: init done 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.965+0000 7fca01c5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.965+0000 7fca01c5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.965+0000 7fca01c5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.965+0000 7fca01c5c640 1 -- 172.21.15.19:0/2398717187 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9fc06a700 con 0x7fc9fc1dbe00 2024-05-04T17:45:25.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85b3fff640 1 -- 172.21.15.19:0/2523051078 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc043070 con 0x7f85cc072050 2024-05-04T17:45:25.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:25.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85b3fff640 10 monclient: dump: 2024-05-04T17:45:25.972 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.972 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.972 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85b3fff640 10 monclient: _finish_auth 0 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85b3fff640 10 monclient: _check_auth_tickets 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85d1c2c640 5 monclient: authenticate success, global_id 4308 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85d1c2c640 10 monclient: _renew_subs 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85d1c2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85d1c2c640 1 -- 172.21.15.19:0/2523051078 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85cc13b5e0 con 0x7f85cc072050 2024-05-04T17:45:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85d1c2c640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85d1c2c640 10 monclient: _renew_subs 2024-05-04T17:45:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85d1c2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85d1c2c640 1 -- 172.21.15.19:0/2523051078 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85cc1485e0 con 0x7f85cc072050 2024-05-04T17:45:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.967+0000 7f85d1c2c640 1 librados: init done 2024-05-04T17:45:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.969+0000 7f85d1c2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.969+0000 7f85d1c2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.969+0000 7f85d1c2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.969+0000 7f85d1c2c640 1 -- 172.21.15.19:0/2523051078 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85cc04f660 con 0x7f85cc072050 2024-05-04T17:45:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.970+0000 7f85b3fff640 1 -- 172.21.15.19:0/2523051078 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85bc02f070 con 0x7f85cc072050 2024-05-04T17:45:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.970+0000 7f85b3fff640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:25.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.970+0000 7f85c9ffb640 4 set_mon_vals no callback set 2024-05-04T17:45:25.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.982+0000 7f85b3fff640 1 -- 172.21.15.19:0/2523051078 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85bc04d060 con 0x7f85cc072050 2024-05-04T17:45:25.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.982+0000 7f85b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:25.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.982+0000 7f85b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:25.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.982+0000 7f85b3fff640 10 monclient: dump: 2024-05-04T17:45:25.983 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:25.983 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.983 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.983 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:25.983 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:25.984 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.984 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:25.984 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:25.984 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:25.984 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.988+0000 7fdb2b360640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.988+0000 7fdb2b360640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.988+0000 7fdb2b360640 10 monclient: monmap: 2024-05-04T17:45:25.989 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.989 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.989 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.989409+0000 2024-05-04T17:45:25.989 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.989409+0000 2024-05-04T17:45:25.989 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.989 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.989 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.990 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.990 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.990 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.990 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.990 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.989+0000 7fdb2b360640 1 Processor -- start 2024-05-04T17:45:25.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.989+0000 7fdb2b360640 1 -- start start 2024-05-04T17:45:25.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.989+0000 7fdb2b360640 10 monclient: init 2024-05-04T17:45:25.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.990+0000 7fdb2b360640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.990+0000 7fdb2b360640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-05-04T17:45:25.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.990+0000 7fdb2b360640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb240a9720 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:25.992 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.990+0000 7fdb2b360640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb240a9b00 0x7fdb240aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.990+0000 7fdb2b360640 10 monclient(hunting): picked mon.noname-c con 0x7fdb240a9b00 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:25.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.991+0000 7fdb2b360640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fdb240b2e90 0x7fdb2413fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.991+0000 7fdb2b360640 10 monclient(hunting): picked mon.noname-a con 0x7fdb240b2e90 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:25.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.991+0000 7fdb2b360640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb240abc40 con 0x7fdb240a9720 2024-05-04T17:45:25.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.991+0000 7fdb2b360640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.991+0000 7fdb2b360640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fdb24146fa0 con 0x7fdb240b2e90 2024-05-04T17:45:25.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.991+0000 7fdb2b360640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.991+0000 7fdb2b360640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdb24146090 con 0x7fdb240a9b00 2024-05-04T17:45:25.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.991+0000 7fdb2b360640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.991+0000 7fdb2b360640 10 monclient(hunting): authenticate will time out at 1348.105009s 2024-05-04T17:45:25.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.993+0000 7f732047b640 10 monclient: get_monmap_and_config 2024-05-04T17:45:25.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.993+0000 7f732047b640 10 monclient: build_initial_monmap 2024-05-04T17:45:25.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.993+0000 7f732047b640 10 monclient: monmap: 2024-05-04T17:45:25.994 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:25.994 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:25.994 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:25.993974+0000 2024-05-04T17:45:25.994 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:25.993974+0000 2024-05-04T17:45:25.995 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:25.995 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:25.995 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:25.995 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:25.995 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:25.995 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:25.995 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:25.995 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:25.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.993+0000 7f732047b640 1 Processor -- start 2024-05-04T17:45:25.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.993+0000 7f732047b640 1 -- start start 2024-05-04T17:45:25.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.993+0000 7f732047b640 10 monclient: init 2024-05-04T17:45:25.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.994+0000 7f732047b640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:25.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.994+0000 7f732047b640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-05-04T17:45:25.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.994+0000 7f732047b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73180a9720 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:25.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.994+0000 7f732047b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73180a9b00 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:25.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.994+0000 7f732047b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73180aa890 0x7f73180b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:25.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.994+0000 7f732047b640 10 monclient(hunting): picked mon.noname-c con 0x7f73180aa890 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:25.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.994+0000 7f732047b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73180abc40 con 0x7f73180a9b00 2024-05-04T17:45:25.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.994+0000 7f732047b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7318146fa0 con 0x7f73180a9720 2024-05-04T17:45:25.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.994+0000 7f732047b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:25.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.994+0000 7f732047b640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f7318146090 con 0x7f73180aa890 2024-05-04T17:45:25.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.994+0000 7f732047b640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:25.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:25.994+0000 7f732047b640 10 monclient(hunting): authenticate will time out at 1348.108623s 2024-05-04T17:45:26.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.008+0000 7fc9e3fff640 1 -- 172.21.15.19:0/2398717187 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc9f401a070 con 0x7fc9fc1dbe00 2024-05-04T17:45:26.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.008+0000 7fc9e3fff640 1 --2- 172.21.15.19:0/2398717187 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc9e806d840 0x7fc9e806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.009+0000 7fc9fb7fe640 1 --2- 172.21.15.19:0/2398717187 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc9e806d840 0x7fc9e806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.009+0000 7fc9fb7fe640 10 monclient: get_auth_request con 0x7fc9e806d840 auth_method 0 2024-05-04T17:45:26.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.009+0000 7fc9fb7fe640 1 --2- 172.21.15.19:0/2398717187 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc9e806d840 0x7fc9e806fd00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0034000 tx=0x7fc9f002f000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.009+0000 7fc9e3fff640 10 client.4305.objecter ms_handle_connect 0x7fc9e806d840 2024-05-04T17:45:26.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.031+0000 7fa213ea5640 10 monclient: get_monmap_and_config 2024-05-04T17:45:26.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.031+0000 7fa213ea5640 10 monclient: build_initial_monmap 2024-05-04T17:45:26.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.031+0000 7fa213ea5640 10 monclient: monmap: 2024-05-04T17:45:26.032 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:26.032 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.032 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:26.032290+0000 2024-05-04T17:45:26.032 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:26.032290+0000 2024-05-04T17:45:26.032 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:26.032 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.033 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:26.033 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:26.033 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:26.033 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:26.033 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:26.033 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.032+0000 7fa213ea5640 1 Processor -- start 2024-05-04T17:45:26.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.032+0000 7fa213ea5640 1 -- start start 2024-05-04T17:45:26.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.032+0000 7fa213ea5640 10 monclient: init 2024-05-04T17:45:26.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa20c0a9720 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa20c0a9b00 0x7fa20c0aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 10 monclient(hunting): picked mon.noname-b con 0x7fa20c0a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa20c0b2e90 0x7fa20c13fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 10 monclient(hunting): picked mon.noname-c con 0x7fa20c0b2e90 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa20c0abc40 con 0x7fa20c0a9720 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa20c146fa0 con 0x7fa20c0b2e90 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fa20c146090 con 0x7fa20c0a9b00 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:26.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.033+0000 7fa213ea5640 10 monclient(hunting): authenticate will time out at 1348.146926s 2024-05-04T17:45:26.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.057+0000 7f6529eac640 10 monclient: get_monmap_and_config 2024-05-04T17:45:26.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.057+0000 7f6529eac640 10 monclient: build_initial_monmap 2024-05-04T17:45:26.058 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.057+0000 7f6529eac640 10 monclient: monmap: 2024-05-04T17:45:26.058 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:26.058 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.058 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:26.058389+0000 2024-05-04T17:45:26.058 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:26.058389+0000 2024-05-04T17:45:26.059 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:26.059 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.059 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:26.059 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:26.059 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:26.059 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:26.059 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:26.059 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.058+0000 7f6529eac640 1 Processor -- start 2024-05-04T17:45:26.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.058+0000 7f6529eac640 1 -- start start 2024-05-04T17:45:26.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.058+0000 7f6529eac640 10 monclient: init 2024-05-04T17:45:26.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.058+0000 7f6529eac640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:26.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.058+0000 7f6529eac640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-05-04T17:45:26.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.058+0000 7f6529eac640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f65240a9fb0 0x7f65240aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.058+0000 7f6529eac640 10 monclient(hunting): picked mon.noname-c con 0x7f65240a9fb0 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:26.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.058+0000 7f6529eac640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65240b27f0 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:26.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.059+0000 7f6529eac640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65241404e0 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:26.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.059+0000 7f6529eac640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65240ab880 con 0x7f65240b27f0 2024-05-04T17:45:26.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.059+0000 7f6529eac640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6524146b00 con 0x7f65241404e0 2024-05-04T17:45:26.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.059+0000 7f6529eac640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.059+0000 7f6529eac640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6524146cd0 con 0x7f65240a9fb0 2024-05-04T17:45:26.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.059+0000 7f6529eac640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:26.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.059+0000 7f6529eac640 10 monclient(hunting): authenticate will time out at 1348.172801s 2024-05-04T17:45:26.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.204+0000 7fd3a97fa640 1 -- 172.21.15.19:0/1129755580 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd3a405a490 con 0x7fd3ac0ab230 2024-05-04T17:45:26.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.204+0000 7fd3a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.204+0000 7fd3a97fa640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:26.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.207+0000 7f731ea18640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73180aa890 0x7f73180b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.208+0000 7f731ea18640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f73180aa890 0x7f73180b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36396/0 (socket says 172.21.15.19:36396) 2024-05-04T17:45:26.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.208+0000 7f731ea18640 1 -- 172.21.15.19:0/3646412379 learned_addr learned my addr 172.21.15.19:0/3646412379 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:26.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.208+0000 7f731ea18640 10 monclient(hunting): get_auth_request con 0x7f73180aa890 auth_method 0 2024-05-04T17:45:26.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.208+0000 7f731ea18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:26.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.208+0000 7f731ea18640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:26.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.208+0000 7f731ea18640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:26.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.209+0000 7f731ea18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:26.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.209+0000 7f731ea18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:26.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.209+0000 7f731ea18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:26.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.209+0000 7fa212442640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa20c0b2e90 0x7fa20c13fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.209+0000 7f65237fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f65240a9fb0 0x7f65240aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.210+0000 7fa212442640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa20c0b2e90 0x7fa20c13fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36402/0 (socket says 172.21.15.19:36402) 2024-05-04T17:45:26.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.210+0000 7fa212442640 1 -- 172.21.15.19:0/3907424556 learned_addr learned my addr 172.21.15.19:0/3907424556 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:26.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.210+0000 7fa212442640 10 monclient(hunting): get_auth_request con 0x7fa20c0b2e90 auth_method 0 2024-05-04T17:45:26.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.210+0000 7fa212442640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:26.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.210+0000 7fa212442640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:26.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.210+0000 7fa212442640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:26.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.210+0000 7f65237fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f65240a9fb0 0x7f65240aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36406/0 (socket says 172.21.15.19:36406) 2024-05-04T17:45:26.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.210+0000 7f65237fe640 1 -- 172.21.15.19:0/3949121342 learned_addr learned my addr 172.21.15.19:0/3949121342 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:26.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.210+0000 7f65237fe640 10 monclient(hunting): get_auth_request con 0x7f65240a9fb0 auth_method 0 2024-05-04T17:45:26.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.210+0000 7f65237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:26.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.210+0000 7f65237fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:26.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.210+0000 7f65237fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:26.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.211+0000 7fa212442640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:26.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.211+0000 7fa212442640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:26.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.211+0000 7fa212442640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:26.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.211+0000 7f65237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:26.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.211+0000 7f65237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:26.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.211+0000 7f65237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:26.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.212+0000 7fdb288fb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb240a9b00 0x7fdb240aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.212+0000 7fa211440640 1 --2- 172.21.15.19:0/3907424556 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa20c0a9b00 0x7fa20c0aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa211440640 10 monclient(hunting): get_auth_request con 0x7fa20c0a9b00 auth_method 0 2024-05-04T17:45:26.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa211440640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:26.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa211440640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:26.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa211440640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:26.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa212442640 10 monclient(hunting): handle_auth_done global_id 4378 payload 274 2024-05-04T17:45:26.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa212442640 1 -- 172.21.15.19:0/3907424556 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa20c0a9b00 msgr2=0x7fa20c0aa890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:26.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa212442640 1 --2- 172.21.15.19:0/3907424556 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa20c0a9b00 0x7fa20c0aa890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa212442640 1 -- 172.21.15.19:0/3907424556 >> v1:172.21.15.19:6789/0 conn(0x7fa20c0a9720 legacy=0x7fa20c146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:26.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa212442640 10 monclient: _finish_hunting 0 2024-05-04T17:45:26.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa212442640 1 monclient: found mon.noname-c 2024-05-04T17:45:26.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa212442640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:26.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa212442640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.213+0000 7fa212442640 1 -- 172.21.15.19:0/3907424556 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20c146370 con 0x7fa20c0b2e90 2024-05-04T17:45:26.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.215+0000 7f65237fe640 10 monclient(hunting): handle_auth_done global_id 4381 payload 274 2024-05-04T17:45:26.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.215+0000 7f65237fe640 1 -- 172.21.15.19:0/3949121342 >> v1:172.21.15.186:6789/0 conn(0x7f65241404e0 legacy=0x7f65241463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:26.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.215+0000 7f65237fe640 1 -- 172.21.15.19:0/3949121342 >> v1:172.21.15.19:6789/0 conn(0x7f65240b27f0 legacy=0x7f652413fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:26.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.215+0000 7f65237fe640 10 monclient: _finish_hunting 0 2024-05-04T17:45:26.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.215+0000 7f65237fe640 1 monclient: found mon.noname-c 2024-05-04T17:45:26.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.215+0000 7f65237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:26.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.215+0000 7f65237fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.215+0000 7f65237fe640 1 -- 172.21.15.19:0/3949121342 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6524146e50 con 0x7f65240a9fb0 2024-05-04T17:45:26.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.216+0000 7fdb288fb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb240a9b00 0x7fdb240aa890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36390/0 (socket says 172.21.15.19:36390) 2024-05-04T17:45:26.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.216+0000 7fdb288fb640 1 -- 172.21.15.19:0/380924399 learned_addr learned my addr 172.21.15.19:0/380924399 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:26.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.216+0000 7fdb288fb640 10 monclient(hunting): get_auth_request con 0x7fdb240a9b00 auth_method 0 2024-05-04T17:45:26.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.216+0000 7fdb288fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:26.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.216+0000 7fdb288fb640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:26.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.216+0000 7fdb288fb640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:26.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.219+0000 7f731ea18640 10 monclient(hunting): handle_auth_done global_id 4375 payload 274 2024-05-04T17:45:26.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.219+0000 7f731ea18640 1 -- 172.21.15.19:0/3646412379 >> v1:172.21.15.186:6789/0 conn(0x7f73180a9720 legacy=0x7f7318146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:26.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.219+0000 7f731ea18640 1 -- 172.21.15.19:0/3646412379 >> v1:172.21.15.19:6789/0 conn(0x7f73180a9b00 legacy=0x7f731813fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:26.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.219+0000 7f731ea18640 10 monclient: _finish_hunting 0 2024-05-04T17:45:26.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.219+0000 7f731ea18640 1 monclient: found mon.noname-c 2024-05-04T17:45:26.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.219+0000 7f731ea18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:26.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.219+0000 7f731ea18640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.219+0000 7f731ea18640 1 -- 172.21.15.19:0/3646412379 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7318146370 con 0x7f73180aa890 2024-05-04T17:45:26.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.219+0000 7fdb288fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:26.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.219+0000 7fdb288fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:26.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.219+0000 7fdb288fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:26.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.223+0000 7fdb288fb640 10 monclient(hunting): handle_auth_done global_id 4384 payload 274 2024-05-04T17:45:26.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.223+0000 7fdb288fb640 1 -- 172.21.15.19:0/380924399 >> v2:172.21.15.19:3300/0 conn(0x7fdb240b2e90 msgr2=0x7fdb2413fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:26.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.223+0000 7fdb288fb640 1 --2- 172.21.15.19:0/380924399 >> v2:172.21.15.19:3300/0 conn(0x7fdb240b2e90 0x7fdb2413fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.223+0000 7fdb288fb640 1 -- 172.21.15.19:0/380924399 >> v1:172.21.15.19:6789/0 conn(0x7fdb240a9720 legacy=0x7fdb24146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:26.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.223+0000 7fdb288fb640 10 monclient: _finish_hunting 0 2024-05-04T17:45:26.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.223+0000 7fdb288fb640 1 monclient: found mon.noname-c 2024-05-04T17:45:26.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.223+0000 7fdb288fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:26.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.223+0000 7fdb288fb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.223+0000 7fdb288fb640 1 -- 172.21.15.19:0/380924399 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb24146370 con 0x7fdb240a9b00 2024-05-04T17:45:26.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.224+0000 7f5a457fa640 1 -- 172.21.15.19:0/2669308439 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5a40032cb0 con 0x7f5a40035680 2024-05-04T17:45:26.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.224+0000 7f5a457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.225 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.224+0000 7f5a457fa640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:26.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.228+0000 7f731ea18640 1 --2- 172.21.15.19:0/3646412379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73180aa890 0x7f73180b2e90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7310009e80 tx=0x7f731002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ade389c9b34ab04d server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.228+0000 7f65237fe640 1 --2- 172.21.15.19:0/3949121342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240a9fb0 0x7f65240aa390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6514009870 tx=0x7f651402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23f31333c7178f8f server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.231 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.230+0000 7fdb288fb640 1 --2- 172.21.15.19:0/380924399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb240a9b00 0x7fdb240aa890 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdb14009870 tx=0x7fdb1402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce94c71dd8e94d10 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.234+0000 7f731d215640 1 -- 172.21.15.19:0/3646412379 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731002ca70 con 0x7f73180aa890 2024-05-04T17:45:26.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.234+0000 7f731d215640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.234+0000 7f731d215640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:26.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.234+0000 7f731d215640 10 monclient: dump: 2024-05-04T17:45:26.235 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.235 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.235 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.235 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.235 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.236 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.236 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.236 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.236 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.236 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.234+0000 7f731d215640 10 monclient: _finish_auth 0 2024-05-04T17:45:26.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.234+0000 7f731d215640 10 monclient: _check_auth_tickets 2024-05-04T17:45:26.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.234+0000 7f731d215640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:26.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.234+0000 7f732047b640 5 monclient: authenticate success, global_id 4375 2024-05-04T17:45:26.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.234+0000 7f732047b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:26.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 1 -- 172.21.15.19:0/380924399 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14035020 con 0x7fdb240a9b00 2024-05-04T17:45:26.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:26.241 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 10 monclient: dump: 2024-05-04T17:45:26.241 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.241 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.241 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.241 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.242 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.242 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.242 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.242 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 10 monclient: _finish_auth 0 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 10 monclient: _check_auth_tickets 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 1 -- 172.21.15.19:0/380924399 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb14031870 con 0x7fdb240a9b00 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb2b360640 5 monclient: authenticate success, global_id 4384 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb2b360640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb2b360640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb2b360640 4 set_mon_vals no callback set 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 1 -- 172.21.15.19:0/380924399 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14031c40 con 0x7fdb240a9b00 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:26.243 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7fdb1bfff640 10 monclient: dump: 2024-05-04T17:45:26.244 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.244 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.244 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.244 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.244 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.244 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.244 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.244 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.244 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.244 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7f731d215640 1 -- 172.21.15.19:0/3646412379 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f731002cc10 con 0x7f73180aa890 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7f731d215640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7f731d215640 1 -- 172.21.15.19:0/3646412379 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73100343f0 con 0x7f73180aa890 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7f732047b640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7f732047b640 4 set_mon_vals no callback set 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7f731d215640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7f731d215640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.240+0000 7f731d215640 10 monclient: dump: 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.245 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.246 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.246 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.246 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.246 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.246 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.246 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7fdb2b360640 10 monclient: shutdown 2024-05-04T17:45:26.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7fdb2b360640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:26.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7fdb2b360640 1 -- 172.21.15.19:0/380924399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb240a9b00 msgr2=0x7fdb240aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7fdb2b360640 1 --2- 172.21.15.19:0/380924399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb240a9b00 0x7fdb240aa890 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdb14009870 tx=0x7fdb1402b910 comp rx=0 tx=0).stop 2024-05-04T17:45:26.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7fdb2b360640 1 -- 172.21.15.19:0/380924399 shutdown_connections 2024-05-04T17:45:26.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7fdb2b360640 1 --2- 172.21.15.19:0/380924399 >> v2:172.21.15.19:3300/0 conn(0x7fdb240b2e90 0x7fdb2413fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7fdb2b360640 1 --2- 172.21.15.19:0/380924399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb240a9b00 0x7fdb240aa890 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7fdb2b360640 1 -- 172.21.15.19:0/380924399 >> 172.21.15.19:0/380924399 conn(0x7fdb24067780 msgr2=0x7fdb240b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:26.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7fdb2b360640 1 -- 172.21.15.19:0/380924399 shutdown_connections 2024-05-04T17:45:26.247 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7fdb2b360640 1 -- 172.21.15.19:0/380924399 wait complete. 2024-05-04T17:45:26.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7f732047b640 10 monclient: shutdown 2024-05-04T17:45:26.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7f732047b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:26.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7f732047b640 1 -- 172.21.15.19:0/3646412379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73180aa890 msgr2=0x7f73180b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7f732047b640 1 --2- 172.21.15.19:0/3646412379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73180aa890 0x7f73180b2e90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7310009e80 tx=0x7f731002bed0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7f732047b640 1 -- 172.21.15.19:0/3646412379 shutdown_connections 2024-05-04T17:45:26.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7f732047b640 1 --2- 172.21.15.19:0/3646412379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73180aa890 0x7f73180b2e90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7f732047b640 1 -- 172.21.15.19:0/3646412379 >> 172.21.15.19:0/3646412379 conn(0x7f7318067780 msgr2=0x7f73180b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:26.248 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.241+0000 7f732047b640 1 -- 172.21.15.19:0/3646412379 shutdown_connections 2024-05-04T17:45:26.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f732047b640 1 -- 172.21.15.19:0/3646412379 wait complete. 2024-05-04T17:45:26.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 1 -- 172.21.15.19:0/3949121342 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651402e070 con 0x7f65240a9fb0 2024-05-04T17:45:26.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:26.249 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 10 monclient: dump: 2024-05-04T17:45:26.249 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.249 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.249 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 10 monclient: _finish_auth 0 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 10 monclient: _check_auth_tickets 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 1 -- 172.21.15.19:0/3949121342 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6514037070 con 0x7f65240a9fb0 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 1 -- 172.21.15.19:0/3949121342 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514031a00 con 0x7f65240a9fb0 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.250 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f65227fc640 10 monclient: dump: 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f6529eac640 5 monclient: authenticate success, global_id 4381 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f6529eac640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:26.251 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7f6529eac640 4 set_mon_vals no callback set 2024-05-04T17:45:26.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7fdb2b360640 10 monclient: build_initial_monmap 2024-05-04T17:45:26.252 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7fdb2b360640 10 monclient: monmap: 2024-05-04T17:45:26.252 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:26.252 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:26.253 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:26.253 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:26.253 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:26.253 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.253 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:26.253 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:26.253 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:26.253 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7fa212442640 1 --2- 172.21.15.19:0/3907424556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa20c0b2e90 0x7fa20c13fea0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa204009e50 tx=0x7fa20402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3a6164eeb70abe5 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7fdb2b360640 1 librados: starting msgr at 2024-05-04T17:45:26.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.242+0000 7fdb2b360640 1 librados: starting objecter 2024-05-04T17:45:26.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7fdb2b360640 1 Processor -- start 2024-05-04T17:45:26.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7fdb2b360640 1 -- start start 2024-05-04T17:45:26.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f732047b640 10 monclient: build_initial_monmap 2024-05-04T17:45:26.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7fdb2b360640 1 librados: setting wanted keys 2024-05-04T17:45:26.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7fdb2b360640 1 librados: calling monclient init 2024-05-04T17:45:26.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7fdb2b360640 10 monclient: init 2024-05-04T17:45:26.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f732047b640 10 monclient: monmap: 2024-05-04T17:45:26.255 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:26.255 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:26.255 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:26.256 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:26.256 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:26.256 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.256 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:26.256 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:26.256 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:26.256 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f6529eac640 10 monclient: shutdown 2024-05-04T17:45:26.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f6529eac640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:26.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f6529eac640 1 -- 172.21.15.19:0/3949121342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240a9fb0 msgr2=0x7f65240aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f6529eac640 1 --2- 172.21.15.19:0/3949121342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240a9fb0 0x7f65240aa390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6514009870 tx=0x7f651402b880 comp rx=0 tx=0).stop 2024-05-04T17:45:26.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f6529eac640 1 -- 172.21.15.19:0/3949121342 shutdown_connections 2024-05-04T17:45:26.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f6529eac640 1 --2- 172.21.15.19:0/3949121342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240a9fb0 0x7f65240aa390 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f6529eac640 1 -- 172.21.15.19:0/3949121342 >> 172.21.15.19:0/3949121342 conn(0x7f65240b4e90 msgr2=0x7f65240b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:26.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f6529eac640 1 -- 172.21.15.19:0/3949121342 shutdown_connections 2024-05-04T17:45:26.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f6529eac640 1 -- 172.21.15.19:0/3949121342 wait complete. 2024-05-04T17:45:26.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f732047b640 1 librados: starting msgr at 2024-05-04T17:45:26.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f732047b640 1 librados: starting objecter 2024-05-04T17:45:26.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f732047b640 1 Processor -- start 2024-05-04T17:45:26.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f732047b640 1 -- start start 2024-05-04T17:45:26.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f732047b640 1 librados: setting wanted keys 2024-05-04T17:45:26.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f732047b640 1 librados: calling monclient init 2024-05-04T17:45:26.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7f732047b640 10 monclient: init 2024-05-04T17:45:26.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7fdb2b360640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:26.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7fdb2b360640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-04T17:45:26.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.243+0000 7fdb2b360640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fdb240a9720 0x7fdb2413ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.260 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 10 monclient(hunting): picked mon.noname-c con 0x7fdb240a9720 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:26.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdb240a9b00 0x7fdb240b7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 10 monclient(hunting): picked mon.noname-a con 0x7fdb240a9b00 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:26.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb240b2e90 0x7fdb24139240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 10 monclient(hunting): picked mon.noname-b con 0x7fdb240b2e90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:26.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fdb24140770 con 0x7fdb240a9b00 2024-05-04T17:45:26.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdb241405f0 con 0x7fdb240b2e90 2024-05-04T17:45:26.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fdb2414d420 con 0x7fdb240a9720 2024-05-04T17:45:26.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:26.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb2b360640 10 monclient(hunting): authenticate will time out at 1348.358158s 2024-05-04T17:45:26.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7f6529eac640 10 monclient: build_initial_monmap 2024-05-04T17:45:26.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7f732047b640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:26.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7f732047b640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-04T17:45:26.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7f6529eac640 10 monclient: monmap: 2024-05-04T17:45:26.262 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:26.262 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:26.262 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:26.262 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:26.263 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:26.263 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.263 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:26.263 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:26.263 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:26.263 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7f732047b640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f73180a9720 0x7f7318143280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.263 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7f732047b640 10 monclient(hunting): picked mon.noname-c con 0x7f73180a9720 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:26.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7f732047b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f73180a9b00 0x7f73181437c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7fdb290fc640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fdb240a9720 0x7fdb2413ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.244+0000 7f732047b640 10 monclient(hunting): picked mon.noname-a con 0x7f73180a9b00 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:26.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f732047b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73180aa890 0x7f7318140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f732047b640 10 monclient(hunting): picked mon.noname-b con 0x7f73180aa890 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:26.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f732047b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f732047b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f731814e5b0 con 0x7f73180a9b00 2024-05-04T17:45:26.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f732047b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f732047b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f731814e430 con 0x7f73180aa890 2024-05-04T17:45:26.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f732047b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f732047b640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f731814e730 con 0x7f73180a9720 2024-05-04T17:45:26.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f732047b640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:26.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f732047b640 10 monclient(hunting): authenticate will time out at 1348.358728s 2024-05-04T17:45:26.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f6529eac640 1 librados: starting msgr at 2024-05-04T17:45:26.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f6529eac640 1 librados: starting objecter 2024-05-04T17:45:26.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f6529eac640 1 Processor -- start 2024-05-04T17:45:26.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f6529eac640 1 -- start start 2024-05-04T17:45:26.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f6529eac640 1 librados: setting wanted keys 2024-05-04T17:45:26.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f6529eac640 1 librados: calling monclient init 2024-05-04T17:45:26.266 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.245+0000 7f6529eac640 10 monclient: init 2024-05-04T17:45:26.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7fdb290fc640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fdb240a9720 0x7fdb2413ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:37082/0 (socket says 172.21.15.19:37082) 2024-05-04T17:45:26.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7fdb290fc640 1 -- 172.21.15.19:0/1990042712 learned_addr learned my addr 172.21.15.19:0/1990042712 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:26.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7fdb290fc640 10 monclient(hunting): get_auth_request con 0x7fdb240a9720 auth_method 0 2024-05-04T17:45:26.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7fdb290fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:26.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7fdb290fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:26.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7fdb290fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:26.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:26.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T17:45:26.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240a9fb0 0x7f6524145760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 10 monclient(hunting): picked mon.noname-b con 0x7f65240a9fb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:26.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65240b27f0 0x7f652406e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 10 monclient(hunting): picked mon.noname-a con 0x7f65240b27f0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:26.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f65241404e0 0x7f6524132c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 10 monclient(hunting): picked mon.noname-c con 0x7f65241404e0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:26.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f652414e7b0 con 0x7f65240b27f0 2024-05-04T17:45:26.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f652414e630 con 0x7f65240a9fb0 2024-05-04T17:45:26.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f652414e930 con 0x7f65241404e0 2024-05-04T17:45:26.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:26.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f6529eac640 10 monclient(hunting): authenticate will time out at 1348.360216s 2024-05-04T17:45:26.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.246+0000 7f731e217640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f73180a9720 0x7f7318143280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fdb290fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:26.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fdb290fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:26.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fdb290fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 1 -- 172.21.15.19:0/3907424556 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20402ca70 con 0x7fa20c0b2e90 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 10 monclient: dump: 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.271 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 10 monclient: _finish_auth 0 2024-05-04T17:45:26.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 10 monclient: _check_auth_tickets 2024-05-04T17:45:26.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:26.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 1 -- 172.21.15.19:0/3907424556 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa20402cc10 con 0x7fa20c0b2e90 2024-05-04T17:45:26.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7f731e217640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f73180a9720 0x7f7318143280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:37094/0 (socket says 172.21.15.19:37094) 2024-05-04T17:45:26.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7f731e217640 1 -- 172.21.15.19:0/3103994419 learned_addr learned my addr 172.21.15.19:0/3103994419 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:26.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7f731e217640 10 monclient(hunting): get_auth_request con 0x7f73180a9720 auth_method 0 2024-05-04T17:45:26.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7f731e217640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:26.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7f731e217640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:26.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7f731e217640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:26.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:26.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 1 -- 172.21.15.19:0/3907424556 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2040343f0 con 0x7fa20c0b2e90 2024-05-04T17:45:26.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:26.274 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.247+0000 7fa210c3f640 10 monclient: dump: 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.248+0000 7fa213ea5640 5 monclient: authenticate success, global_id 4378 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.248+0000 7fa213ea5640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.248+0000 7fa213ea5640 4 set_mon_vals no callback set 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.248+0000 7fa213ea5640 10 monclient: shutdown 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.248+0000 7fa213ea5640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.248+0000 7fa213ea5640 1 -- 172.21.15.19:0/3907424556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa20c0b2e90 msgr2=0x7fa20c13fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.275 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.248+0000 7fa213ea5640 1 --2- 172.21.15.19:0/3907424556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa20c0b2e90 0x7fa20c13fea0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa204009e50 tx=0x7fa20402bed0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.248+0000 7fa213ea5640 1 -- 172.21.15.19:0/3907424556 shutdown_connections 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.248+0000 7fa213ea5640 1 --2- 172.21.15.19:0/3907424556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa20c0b2e90 0x7fa20c13fea0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.248+0000 7fa213ea5640 1 --2- 172.21.15.19:0/3907424556 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa20c0a9b00 0x7fa20c0aa890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.248+0000 7fa213ea5640 1 -- 172.21.15.19:0/3907424556 >> 172.21.15.19:0/3907424556 conn(0x7fa20c067780 msgr2=0x7fa20c0b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.249+0000 7fa213ea5640 1 -- 172.21.15.19:0/3907424556 shutdown_connections 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.249+0000 7fa213ea5640 1 -- 172.21.15.19:0/3907424556 wait complete. 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.249+0000 7fa213ea5640 10 monclient: build_initial_monmap 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.249+0000 7fa213ea5640 10 monclient: monmap: 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:26.276 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:26.277 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:26.277 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.250+0000 7fa213ea5640 1 librados: starting msgr at 2024-05-04T17:45:26.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.250+0000 7fa213ea5640 1 librados: starting objecter 2024-05-04T17:45:26.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.250+0000 7fa213ea5640 1 Processor -- start 2024-05-04T17:45:26.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.250+0000 7fa213ea5640 1 -- start start 2024-05-04T17:45:26.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.250+0000 7fa213ea5640 1 librados: setting wanted keys 2024-05-04T17:45:26.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.250+0000 7fa213ea5640 1 librados: calling monclient init 2024-05-04T17:45:26.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.250+0000 7fa213ea5640 10 monclient: init 2024-05-04T17:45:26.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:26.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-04T17:45:26.277 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fdb298fd640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb240b2e90 0x7fdb24139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa20c0a9720 0x7fa20c0ac960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 10 monclient(hunting): picked mon.noname-b con 0x7fa20c0a9720 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:26.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa20c0a9b00 0x7fa20c144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 10 monclient(hunting): picked mon.noname-c con 0x7fa20c0a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:26.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7f731ea18640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73180aa890 0x7f7318140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa20c0b2e90 0x7fa20c144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 10 monclient(hunting): picked mon.noname-a con 0x7fa20c0b2e90 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:26.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fa20c140770 con 0x7fa20c0b2e90 2024-05-04T17:45:26.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa20c1405f0 con 0x7fa20c0a9720 2024-05-04T17:45:26.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:26.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fa20c14d420 con 0x7fa20c0a9b00 2024-05-04T17:45:26.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:26.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.251+0000 7fa213ea5640 10 monclient(hunting): authenticate will time out at 1348.365023s 2024-05-04T17:45:26.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb298fd640 10 monclient(hunting): get_auth_request con 0x7fdb240b2e90 auth_method 0 2024-05-04T17:45:26.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb298fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:26.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb298fd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:26.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb298fd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:26.280 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7f731ea18640 10 monclient(hunting): get_auth_request con 0x7f73180aa890 auth_method 0 2024-05-04T17:45:26.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7f731ea18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:26.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7f731ea18640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:26.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7f731ea18640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:26.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb290fc640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-05-04T17:45:26.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb290fc640 1 -- 172.21.15.19:0/1990042712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb240b2e90 msgr2=0x7fdb24139240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb290fc640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb240b2e90 0x7fdb24139240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb290fc640 1 -- 172.21.15.19:0/1990042712 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdb240a9b00 msgr2=0x7fdb240b7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb290fc640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdb240a9b00 0x7fdb240b7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb290fc640 10 monclient: _finish_hunting 0 2024-05-04T17:45:26.281 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb290fc640 1 monclient: found mon.noname-c 2024-05-04T17:45:26.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb290fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:26.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb290fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T17:45:26.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7fdb290fc640 1 -- 172.21.15.19:0/1990042712 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb24139780 con 0x7fdb240a9720 2024-05-04T17:45:26.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.252+0000 7f65237fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240a9fb0 0x7f6524145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.253+0000 7f65237fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240a9fb0 0x7f6524145760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36432/0 (socket says 172.21.15.19:36432) 2024-05-04T17:45:26.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.253+0000 7f65237fe640 1 -- 172.21.15.19:0/4122109576 learned_addr learned my addr 172.21.15.19:0/4122109576 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:26.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.253+0000 7f65237fe640 10 monclient(hunting): get_auth_request con 0x7f65240a9fb0 auth_method 0 2024-05-04T17:45:26.282 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.253+0000 7f65237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:26.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.253+0000 7f65237fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:26.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.253+0000 7f65237fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:26.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.253+0000 7f731ea18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:26.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.253+0000 7f731ea18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:26.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.253+0000 7f731ea18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:26.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.254+0000 7f65237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:26.283 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.254+0000 7f65237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:26.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.254+0000 7f65237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:26.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.255+0000 7f731e217640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:26.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.255+0000 7f731e217640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:26.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.255+0000 7f731e217640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:26.284 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.257+0000 7f731ea18640 10 monclient(hunting): handle_auth_done global_id 4390 payload 824 2024-05-04T17:45:26.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.257+0000 7f731ea18640 1 -- 172.21.15.19:0/3103994419 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f73180a9720 msgr2=0x7f7318143280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.257+0000 7f731ea18640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f73180a9720 0x7f7318143280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.257+0000 7f731ea18640 1 -- 172.21.15.19:0/3103994419 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f73180a9b00 msgr2=0x7f73181437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.257+0000 7f731ea18640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f73180a9b00 0x7f73181437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.257+0000 7f731ea18640 10 monclient: _finish_hunting 0 2024-05-04T17:45:26.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.257+0000 7f731ea18640 1 monclient: found mon.noname-b 2024-05-04T17:45:26.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.257+0000 7f731ea18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:26.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.257+0000 7f731ea18640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.257+0000 7f731ea18640 1 -- 172.21.15.19:0/3103994419 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7318142810 con 0x7f73180aa890 2024-05-04T17:45:26.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.260+0000 7f65237fe640 10 monclient(hunting): handle_auth_done global_id 4393 payload 824 2024-05-04T17:45:26.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.260+0000 7f65237fe640 1 -- 172.21.15.19:0/4122109576 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f65241404e0 msgr2=0x7f6524132c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.260+0000 7f65237fe640 1 --2- 172.21.15.19:0/4122109576 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f65241404e0 0x7f6524132c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.260+0000 7f65237fe640 1 -- 172.21.15.19:0/4122109576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65240b27f0 msgr2=0x7f652406e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.260+0000 7f65237fe640 1 --2- 172.21.15.19:0/4122109576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65240b27f0 0x7f652406e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.260+0000 7f65237fe640 10 monclient: _finish_hunting 0 2024-05-04T17:45:26.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.260+0000 7f65237fe640 1 monclient: found mon.noname-b 2024-05-04T17:45:26.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.260+0000 7f65237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:26.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.260+0000 7f65237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.260+0000 7f65237fe640 1 -- 172.21.15.19:0/4122109576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65241431a0 con 0x7f65240a9fb0 2024-05-04T17:45:26.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.264+0000 7fdb290fc640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fdb240a9720 0x7fdb2413ca40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000c410 tx=0x7fdb2000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.264+0000 7fdb19ffb640 10 client.?.objecter ms_handle_connect 0x7fdb240a9720 2024-05-04T17:45:26.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.264+0000 7fdb19ffb640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:26.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.264+0000 7f65237fe640 1 --2- 172.21.15.19:0/4122109576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240a9fb0 0x7f6524145760 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6514009990 tx=0x7f65140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.264+0000 7f650bfff640 10 client.?.objecter ms_handle_connect 0x7f65240a9fb0 2024-05-04T17:45:26.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.264+0000 7f650bfff640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:26.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.265+0000 7f731ea18640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73180aa890 0x7f7318140ca0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f73180b91d0 tx=0x7f7310004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.265+0000 7f730effd640 10 client.?.objecter ms_handle_connect 0x7f73180aa890 2024-05-04T17:45:26.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.265+0000 7f730effd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:26.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.266+0000 7fa211c41640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa20c0a9720 0x7fa20c0ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.269+0000 7fa211c41640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa20c0a9720 0x7fa20c0ac960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36442/0 (socket says 172.21.15.19:36442) 2024-05-04T17:45:26.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.269+0000 7fa211c41640 1 -- 172.21.15.19:0/2373148421 learned_addr learned my addr 172.21.15.19:0/2373148421 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:26.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.269+0000 7fa211c41640 10 monclient(hunting): get_auth_request con 0x7fa20c0a9720 auth_method 0 2024-05-04T17:45:26.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.269+0000 7fa211c41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:26.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.269+0000 7fa211c41640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:26.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.269+0000 7fa211c41640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:26.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.270+0000 7fa211c41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:26.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.270+0000 7fa211c41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:26.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.270+0000 7fa211c41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:26.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f650bfff640 1 -- 172.21.15.19:0/4122109576 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514032a40 con 0x7f65240a9fb0 2024-05-04T17:45:26.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f650bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f650bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f650bfff640 10 monclient: dump: 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f650bfff640 10 monclient: _finish_auth 0 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f650bfff640 10 monclient: _check_auth_tickets 2024-05-04T17:45:26.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f650bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f6529eac640 5 monclient: authenticate success, global_id 4393 2024-05-04T17:45:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f6529eac640 10 monclient: _renew_subs 2024-05-04T17:45:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f6529eac640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f6529eac640 1 -- 172.21.15.19:0/4122109576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6524143480 con 0x7f65240a9fb0 2024-05-04T17:45:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f6529eac640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f6529eac640 10 monclient: _renew_subs 2024-05-04T17:45:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f6529eac640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f6529eac640 1 -- 172.21.15.19:0/4122109576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65241331c0 con 0x7f65240a9fb0 2024-05-04T17:45:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7f6529eac640 1 librados: init done 2024-05-04T17:45:26.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.271+0000 7fa211440640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa20c0a9b00 0x7fa20c144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fa211440640 10 monclient(hunting): get_auth_request con 0x7fa20c0a9b00 auth_method 0 2024-05-04T17:45:26.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fa211440640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:26.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fa211440640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:26.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fa211440640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:26.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7f6529eac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7f6529eac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7f6529eac640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7f6529eac640 1 -- 172.21.15.19:0/4122109576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6524068b10 con 0x7f65240a9fb0 2024-05-04T17:45:26.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 1 -- 172.21.15.19:0/1990042712 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20017070 con 0x7fdb240a9720 2024-05-04T17:45:26.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:26.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 10 monclient: dump: 2024-05-04T17:45:26.295 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.295 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.295 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.295 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.295 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.295 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.295 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 10 monclient: _finish_auth 0 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 10 monclient: _check_auth_tickets 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 1 -- 172.21.15.19:0/1990042712 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb200035f0 con 0x7fdb240a9720 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 1 -- 172.21.15.19:0/1990042712 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb200039e0 con 0x7fdb240a9720 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.272+0000 7fdb19ffb640 10 monclient: dump: 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.296 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.273+0000 7fdb1b7fe640 4 set_mon_vals no callback set 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.273+0000 7fdb2b360640 5 monclient: authenticate success, global_id 4286 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.273+0000 7fdb2b360640 10 monclient: _renew_subs 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.273+0000 7fdb2b360640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.273+0000 7fdb2b360640 1 -- 172.21.15.19:0/1990042712 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb241385d0 con 0x7fdb240a9720 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.273+0000 7fdb2b360640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.273+0000 7fdb2b360640 10 monclient: _renew_subs 2024-05-04T17:45:26.297 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.273+0000 7fdb2b360640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.273+0000 7fdb2b360640 1 -- 172.21.15.19:0/1990042712 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb24138cd0 con 0x7fdb240a9720 2024-05-04T17:45:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.273+0000 7fdb2b360640 1 librados: init done 2024-05-04T17:45:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211440640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211440640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211440640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211c41640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-05-04T17:45:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211c41640 1 -- 172.21.15.19:0/2373148421 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa20c0a9b00 msgr2=0x7fa20c144030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211c41640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa20c0a9b00 0x7fa20c144030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.298 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211c41640 1 -- 172.21.15.19:0/2373148421 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa20c0b2e90 msgr2=0x7fa20c144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211c41640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa20c0b2e90 0x7fa20c144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211c41640 10 monclient: _finish_hunting 0 2024-05-04T17:45:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211c41640 1 monclient: found mon.noname-b 2024-05-04T17:45:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211c41640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211c41640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fa211c41640 1 -- 172.21.15.19:0/2373148421 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20c0ae520 con 0x7fa20c0a9720 2024-05-04T17:45:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fdb2b360640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fdb2b360640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fdb2b360640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:26.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.274+0000 7fdb2b360640 1 -- 172.21.15.19:0/1990042712 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb2413ab00 con 0x7fdb240a9720 2024-05-04T17:45:26.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.275+0000 7f650bfff640 1 -- 172.21.15.19:0/4122109576 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6514037020 con 0x7f65240a9fb0 2024-05-04T17:45:26.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.275+0000 7f650bfff640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:26.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.275+0000 7f650bfff640 1 -- 172.21.15.19:0/4122109576 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514053060 con 0x7f65240a9fb0 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.275+0000 7f650bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.275+0000 7f650bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.275+0000 7f650bfff640 10 monclient: dump: 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.275+0000 7f6521ffb640 4 set_mon_vals no callback set 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 1 -- 172.21.15.19:0/3103994419 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7310043020 con 0x7f73180aa890 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 10 monclient: dump: 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.303 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 10 monclient: _finish_auth 0 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 10 monclient: _check_auth_tickets 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 1 -- 172.21.15.19:0/3103994419 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f731003e450 con 0x7f73180aa890 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f732047b640 5 monclient: authenticate success, global_id 4390 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f732047b640 10 monclient: _renew_subs 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f732047b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f732047b640 1 -- 172.21.15.19:0/3103994419 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73181345d0 con 0x7f73180aa890 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 1 -- 172.21.15.19:0/3103994419 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7310009510 con 0x7f73180aa890 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f730effd640 10 monclient: dump: 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f731ca14640 4 set_mon_vals no callback set 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f732047b640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f732047b640 10 monclient: _renew_subs 2024-05-04T17:45:26.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f732047b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f732047b640 1 -- 172.21.15.19:0/3103994419 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73181347f0 con 0x7f73180aa890 2024-05-04T17:45:26.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.278+0000 7f732047b640 1 librados: init done 2024-05-04T17:45:26.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.280+0000 7f732047b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.280+0000 7f732047b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.280+0000 7f732047b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.280+0000 7f732047b640 1 -- 172.21.15.19:0/3103994419 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f731813ac90 con 0x7f73180aa890 2024-05-04T17:45:26.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.304+0000 7fdb19ffb640 1 -- 172.21.15.19:0/1990042712 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdb20003b80 con 0x7fdb240a9720 2024-05-04T17:45:26.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.304+0000 7fdb19ffb640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fdb0806d7f0 0x7fdb0806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.305+0000 7fdb288fb640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fdb0806d7f0 0x7fdb0806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.305+0000 7fdb288fb640 10 monclient: get_auth_request con 0x7fdb0806d7f0 auth_method 0 2024-05-04T17:45:26.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.305+0000 7fdb19ffb640 10 client.4286.objecter ms_handle_connect 0x7fdb0806d7f0 2024-05-04T17:45:26.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.305+0000 7fdb288fb640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fdb0806d7f0 0x7fdb0806fcb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdb14036000 tx=0x7fdb14030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.379+0000 7fdb19ffb640 1 -- 172.21.15.19:0/1990042712 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fdb20056670 con 0x7fdb240a9720 2024-05-04T17:45:26.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.379+0000 7fdb19ffb640 10 client.4286.objecter ms_dispatch 0x7fdb24000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:26.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.379+0000 7fdb19ffb640 3 client.4286.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:26.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.379+0000 7fdb19ffb640 3 client.4286.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:26.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.379+0000 7fdb19ffb640 20 client.4286.objecter dump_active .. 0 homeless 2024-05-04T17:45:26.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.426+0000 7f5a4dad0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-04T17:45:26.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.426+0000 7f5a4dad0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-04T17:45:26.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.426+0000 7f5a4dad0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:26.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.426+0000 7f5a4dad0640 1 -- 172.21.15.19:0/2669308439 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f5a40005710 con 0x7f5a40035680 2024-05-04T17:45:26.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.434+0000 7fd3b2f15640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-04T17:45:26.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.434+0000 7fd3b2f15640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-04T17:45:26.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.434+0000 7fd3b2f15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.434+0000 7fd3b2f15640 1 -- 172.21.15.19:0/1129755580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd3ac0af8d0 con 0x7fd3ac0ab230 2024-05-04T17:45:26.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.526+0000 7f650bfff640 1 -- 172.21.15.19:0/4122109576 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f65140453f0 con 0x7f65240a9fb0 2024-05-04T17:45:26.527 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.527+0000 7f650bfff640 1 --2- 172.21.15.19:0/4122109576 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f651006d7c0 0x7f651006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.527+0000 7f6522ffd640 1 --2- 172.21.15.19:0/4122109576 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f651006d7c0 0x7f651006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.527+0000 7f6522ffd640 10 monclient: get_auth_request con 0x7f651006d7c0 auth_method 0 2024-05-04T17:45:26.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.528+0000 7f6522ffd640 1 --2- 172.21.15.19:0/4122109576 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f651006d7c0 0x7f651006fc80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f65240b3390 tx=0x7f6518007990 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.528 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.528+0000 7f650bfff640 10 client.4393.objecter ms_handle_connect 0x7f651006d7c0 2024-05-04T17:45:26.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.530+0000 7f650bfff640 1 -- 172.21.15.19:0/4122109576 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f6514032de0 con 0x7f65240a9fb0 2024-05-04T17:45:26.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.530+0000 7f650bfff640 10 client.4393.objecter ms_dispatch 0x7f6524000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:26.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.530+0000 7f650bfff640 3 client.4393.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:26.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.530+0000 7f650bfff640 3 client.4393.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:26.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.530+0000 7f650bfff640 20 client.4393.objecter dump_active .. 0 homeless 2024-05-04T17:45:26.540 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.540+0000 7f650bfff640 1 -- 172.21.15.19:0/4122109576 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6514040070 con 0x7f65240a9fb0 2024-05-04T17:45:26.541 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.540+0000 7f650bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.541 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.540+0000 7f650bfff640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:26.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.541+0000 7fa211c41640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa20c0a9720 0x7fa20c0ac960 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa20800b3e0 tx=0x7fa20800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.541+0000 7fa2027fc640 10 client.?.objecter ms_handle_connect 0x7fa20c0a9720 2024-05-04T17:45:26.542 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.541+0000 7fa2027fc640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:26.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.542+0000 7fc9e3fff640 1 -- 172.21.15.19:0/2398717187 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fc9f4057100 con 0x7fc9fc1dbe00 2024-05-04T17:45:26.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.542+0000 7fc9e3fff640 10 client.4305.objecter ms_dispatch 0x7fc9fc000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:26.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.542+0000 7fc9e3fff640 3 client.4305.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:26.543 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.542+0000 7fc9e3fff640 3 client.4305.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:26.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.542+0000 7fc9e3fff640 20 client.4305.objecter dump_active .. 0 homeless 2024-05-04T17:45:26.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.543+0000 7fc9e3fff640 1 -- 172.21.15.19:0/2398717187 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc9f4021020 con 0x7fc9fc1dbe00 2024-05-04T17:45:26.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.543+0000 7fc9e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.544 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.543+0000 7fc9e3fff640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:26.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.565+0000 7fd3a97fa640 1 -- 172.21.15.19:0/1129755580 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd3a4059d90 con 0x7fd3ac0ab230 2024-05-04T17:45:26.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.565+0000 7fd3a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-04T17:45:26.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.565+0000 7fd3a97fa640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:26.567 INFO:teuthology.orchestra.run.smithi019.stdout:81604378632 2024-05-04T17:45:26.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 10 librados: watch_flush enter 2024-05-04T17:45:26.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 10 librados: watch_flush exit 2024-05-04T17:45:26.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 20 client.4369.objecter shutdown clearing up homeless session... 2024-05-04T17:45:26.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 10 client.4369.objecter successfully canceled tick 2024-05-04T17:45:26.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 -- 172.21.15.19:0/1129755580 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fd39806dbc0 msgr2=0x7fd398070080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 --2- 172.21.15.19:0/1129755580 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fd39806dbc0 0x7fd398070080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009a40 tx=0x7fd3a00091c0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 10 monclient: shutdown 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 -- 172.21.15.19:0/1129755580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac0ab230 msgr2=0x7fd3ac141c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 --2- 172.21.15.19:0/1129755580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac0ab230 0x7fd3ac141c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4007d80 tx=0x7fd3a400e4b0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 -- 172.21.15.19:0/1129755580 shutdown_connections 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 --2- 172.21.15.19:0/1129755580 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fd39806dbc0 0x7fd398070080 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 --2- 172.21.15.19:0/1129755580 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd3ac0acb00 0x7fd3ac1e8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 --2- 172.21.15.19:0/1129755580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3ac0ab230 0x7fd3ac141c30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 --2- 172.21.15.19:0/1129755580 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd3ac0aa140 0x7fd3ac145fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 -- 172.21.15.19:0/1129755580 >> 172.21.15.19:0/1129755580 conn(0x7fd3ac0b43a0 msgr2=0x7fd3ac074a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 -- 172.21.15.19:0/1129755580 shutdown_connections 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 -- 172.21.15.19:0/1129755580 wait complete. 2024-05-04T17:45:26.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.568+0000 7fd3b2f15640 1 librados: shutdown 2024-05-04T17:45:26.584 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378633 got 81604378632 for osd.4 2024-05-04T17:45:26.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.618+0000 7fdb19ffb640 1 -- 172.21.15.19:0/1990042712 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdb2413ab00 con 0x7fdb240a9720 2024-05-04T17:45:26.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.618+0000 7fdb19ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.618+0000 7fdb19ffb640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:26.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.628+0000 7f5a457fa640 1 -- 172.21.15.19:0/2669308439 <== mon.2 v2:172.21.15.19:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5a3c017020 con 0x7f5a40035680 2024-05-04T17:45:26.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.628+0000 7f5a457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-04T17:45:26.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.628+0000 7f5a457fa640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:26.629 INFO:teuthology.orchestra.run.smithi019.stdout:68719476742 2024-05-04T17:45:26.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 10 librados: watch_flush enter 2024-05-04T17:45:26.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 10 librados: watch_flush exit 2024-05-04T17:45:26.631 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 20 client.4280.objecter shutdown clearing up homeless session... 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 10 client.4280.objecter successfully canceled tick 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 -- 172.21.15.19:0/2669308439 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5a2c06d810 msgr2=0x7f5a2c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5a2c06d810 0x7f5a2c06fcd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5a34036000 tx=0x7f5a3403c040 comp rx=0 tx=0).stop 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 10 monclient: shutdown 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 -- 172.21.15.19:0/2669308439 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 msgr2=0x7f5a4002fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 0x7f5a4002fe80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00e7e0 tx=0x7f5a3c00eca0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 -- 172.21.15.19:0/2669308439 shutdown_connections 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5a2c06d810 0x7f5a2c06fcd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a400cc640 0x7f5a40033890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5a40039020 0x7f5a40033350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 --2- 172.21.15.19:0/2669308439 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5a40035680 0x7f5a4002fe80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 -- 172.21.15.19:0/2669308439 >> 172.21.15.19:0/2669308439 conn(0x7f5a40006610 msgr2=0x7f5a400c7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:26.632 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 -- 172.21.15.19:0/2669308439 shutdown_connections 2024-05-04T17:45:26.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 -- 172.21.15.19:0/2669308439 wait complete. 2024-05-04T17:45:26.633 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.630+0000 7f5a4dad0640 1 librados: shutdown 2024-05-04T17:45:26.644 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476744 got 68719476742 for osd.3 2024-05-04T17:45:26.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.644+0000 7f85b3fff640 1 -- 172.21.15.19:0/2523051078 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f85bc046070 con 0x7f85cc072050 2024-05-04T17:45:26.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.645+0000 7f85b3fff640 1 --2- 172.21.15.19:0/2523051078 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f85b806d7c0 0x7f85b806fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.645+0000 7f85caffd640 1 --2- 172.21.15.19:0/2523051078 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f85b806d7c0 0x7f85b806fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.645+0000 7f85caffd640 10 monclient: get_auth_request con 0x7f85b806d7c0 auth_method 0 2024-05-04T17:45:26.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.646+0000 7f85caffd640 1 --2- 172.21.15.19:0/2523051078 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f85b806d7c0 0x7f85b806fc80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f85cc06dff0 tx=0x7f85c0006ba0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.646+0000 7f85b3fff640 10 client.4308.objecter ms_handle_connect 0x7f85b806d7c0 2024-05-04T17:45:26.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.678+0000 7f85b3fff640 1 -- 172.21.15.19:0/2523051078 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f85bc079410 con 0x7f85cc072050 2024-05-04T17:45:26.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.678+0000 7f85b3fff640 10 client.4308.objecter ms_dispatch 0x7f85cc000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:26.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.678+0000 7f85b3fff640 3 client.4308.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:26.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.678+0000 7f85b3fff640 3 client.4308.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:26.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.678+0000 7f85b3fff640 20 client.4308.objecter dump_active .. 0 homeless 2024-05-04T17:45:26.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.678+0000 7f85b3fff640 1 -- 172.21.15.19:0/2523051078 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f85bc07ba70 con 0x7f85cc072050 2024-05-04T17:45:26.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.678+0000 7f85b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.678+0000 7f85b3fff640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:26.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa2027fc640 1 -- 172.21.15.19:0/2373148421 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208015020 con 0x7fa20c0a9720 2024-05-04T17:45:26.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa2027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa2027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:26.778 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa2027fc640 10 monclient: dump: 2024-05-04T17:45:26.778 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.778 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.778 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.778 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.778 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.779 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.779 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.779 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.779 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.779 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa2027fc640 10 monclient: _finish_auth 0 2024-05-04T17:45:26.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa2027fc640 10 monclient: _check_auth_tickets 2024-05-04T17:45:26.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa2027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:26.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa2027fc640 1 -- 172.21.15.19:0/2373148421 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa208011850 con 0x7fa20c0a9720 2024-05-04T17:45:26.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa213ea5640 5 monclient: authenticate success, global_id 4396 2024-05-04T17:45:26.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa213ea5640 10 monclient: _renew_subs 2024-05-04T17:45:26.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa213ea5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa213ea5640 1 -- 172.21.15.19:0/2373148421 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa20c0ae800 con 0x7fa20c0a9720 2024-05-04T17:45:26.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa2027fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:26.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa203fff640 4 set_mon_vals no callback set 2024-05-04T17:45:26.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa213ea5640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:26.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa213ea5640 10 monclient: _renew_subs 2024-05-04T17:45:26.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa213ea5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa213ea5640 1 -- 172.21.15.19:0/2373148421 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa20c144ab0 con 0x7fa20c0a9720 2024-05-04T17:45:26.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.777+0000 7fa213ea5640 1 librados: init done 2024-05-04T17:45:26.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.778+0000 7fa213ea5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.778+0000 7fa213ea5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.778+0000 7fa213ea5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.778+0000 7fa213ea5640 1 -- 172.21.15.19:0/2373148421 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa20c13ea40 con 0x7fa20c0a9720 2024-05-04T17:45:26.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.784+0000 7fca01c5c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-04T17:45:26.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.784+0000 7fca01c5c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-04T17:45:26.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.784+0000 7fca01c5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:26.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.784+0000 7fca01c5c640 1 -- 172.21.15.19:0/2398717187 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc9fc073120 con 0x7fc9fc1dbe00 2024-05-04T17:45:26.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.792+0000 7fc9e3fff640 1 -- 172.21.15.19:0/2398717187 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc9f4059760 con 0x7fc9fc1dbe00 2024-05-04T17:45:26.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.793+0000 7fc9e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-04T17:45:26.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.793+0000 7fc9e3fff640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:26.794 INFO:teuthology.orchestra.run.smithi019.stdout:55834574853 2024-05-04T17:45:26.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 10 librados: watch_flush enter 2024-05-04T17:45:26.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 10 librados: watch_flush exit 2024-05-04T17:45:26.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 20 client.4305.objecter shutdown clearing up homeless session... 2024-05-04T17:45:26.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 10 client.4305.objecter successfully canceled tick 2024-05-04T17:45:26.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 -- 172.21.15.19:0/2398717187 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc9e806d840 msgr2=0x7fc9e806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 --2- 172.21.15.19:0/2398717187 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc9e806d840 0x7fc9e806fd00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0034000 tx=0x7fc9f002f000 comp rx=0 tx=0).stop 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 10 monclient: shutdown 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 -- 172.21.15.19:0/2398717187 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9fc1dbe00 msgr2=0x7fc9fc139a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 --2- 172.21.15.19:0/2398717187 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9fc1dbe00 0x7fc9fc139a90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc0b3270 tx=0x7fc9f400cf70 comp rx=0 tx=0).stop 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 -- 172.21.15.19:0/2398717187 shutdown_connections 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 --2- 172.21.15.19:0/2398717187 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc9e806d840 0x7fc9e806fd00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 --2- 172.21.15.19:0/2398717187 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc9fc1dbe00 0x7fc9fc139a90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 --2- 172.21.15.19:0/2398717187 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc9fc141790 0x7fc9fc139550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 --2- 172.21.15.19:0/2398717187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9e4003b40 0x7fc9fc1db8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 -- 172.21.15.19:0/2398717187 >> 172.21.15.19:0/2398717187 conn(0x7fc9fc067780 msgr2=0x7fc9fc13f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 -- 172.21.15.19:0/2398717187 shutdown_connections 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 -- 172.21.15.19:0/2398717187 wait complete. 2024-05-04T17:45:26.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.795+0000 7fca01c5c640 1 librados: shutdown 2024-05-04T17:45:26.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.798+0000 7f6529eac640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-04T17:45:26.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.798+0000 7f6529eac640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-04T17:45:26.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.798+0000 7f6529eac640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:26.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.798+0000 7f6529eac640 1 -- 172.21.15.19:0/4122109576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f65240b30e0 con 0x7f65240a9fb0 2024-05-04T17:45:26.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.810+0000 7fa2027fc640 1 -- 172.21.15.19:0/2373148421 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208011c00 con 0x7fa20c0a9720 2024-05-04T17:45:26.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.810+0000 7fa2027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:26.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.810+0000 7fa2027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:26.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.810+0000 7fa2027fc640 10 monclient: dump: 2024-05-04T17:45:26.811 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:26.811 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:26.811 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.811 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:26.811 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:26.811 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:26.811 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:26.811 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:26.811 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:26.811 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:26.812 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-05-04T17:45:26.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.824+0000 7f730effd640 1 -- 172.21.15.19:0/3103994419 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7310046020 con 0x7f73180aa890 2024-05-04T17:45:26.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.824+0000 7f730effd640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f72fc06d7c0 0x7f72fc06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:26.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.825+0000 7f731e217640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f72fc06d7c0 0x7f72fc06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:26.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.825+0000 7f731e217640 10 monclient: get_auth_request con 0x7f72fc06d7c0 auth_method 0 2024-05-04T17:45:26.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.825+0000 7f730effd640 10 client.4390.objecter ms_handle_connect 0x7f72fc06d7c0 2024-05-04T17:45:26.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.825+0000 7f731e217640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f72fc06d7c0 0x7f72fc06fc80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7314005020 tx=0x7f7314007ef0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:26.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.828+0000 7f730effd640 1 -- 172.21.15.19:0/3103994419 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f7310079dd0 con 0x7f73180aa890 2024-05-04T17:45:26.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.828+0000 7f730effd640 10 client.4390.objecter ms_dispatch 0x7f7318000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:26.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.828+0000 7f730effd640 3 client.4390.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:26.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.828+0000 7f730effd640 3 client.4390.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:26.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.828+0000 7f730effd640 20 client.4390.objecter dump_active .. 0 homeless 2024-05-04T17:45:26.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.840+0000 7f730effd640 1 -- 172.21.15.19:0/3103994419 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f731813ac90 con 0x7f73180aa890 2024-05-04T17:45:26.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.840+0000 7f730effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:26.840 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.840+0000 7f730effd640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:26.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.850+0000 7fdb2b360640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-04T17:45:26.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.850+0000 7fdb2b360640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-04T17:45:26.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.850+0000 7fdb2b360640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:26.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.850+0000 7fdb2b360640 1 -- 172.21.15.19:0/1990042712 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fdb2406a700 con 0x7fdb240a9720 2024-05-04T17:45:26.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.855+0000 7fdb19ffb640 1 -- 172.21.15.19:0/1990042712 <== mon.2 v2:172.21.15.19:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdb20058cd0 con 0x7fdb240a9720 2024-05-04T17:45:26.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.855+0000 7fdb19ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-04T17:45:26.855 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.855+0000 7fdb19ffb640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:26.856 INFO:teuthology.orchestra.run.smithi019.stdout:85899345928 2024-05-04T17:45:26.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 10 librados: watch_flush enter 2024-05-04T17:45:26.858 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 10 librados: watch_flush exit 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 20 client.4286.objecter shutdown clearing up homeless session... 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 10 client.4286.objecter successfully canceled tick 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 -- 172.21.15.19:0/1990042712 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fdb0806d7f0 msgr2=0x7fdb0806fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fdb0806d7f0 0x7fdb0806fcb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdb14036000 tx=0x7fdb14030000 comp rx=0 tx=0).stop 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 10 monclient: shutdown 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 -- 172.21.15.19:0/1990042712 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fdb240a9720 msgr2=0x7fdb2413ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fdb240a9720 0x7fdb2413ca40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000c410 tx=0x7fdb2000c8d0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 -- 172.21.15.19:0/1990042712 shutdown_connections 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fdb0806d7f0 0x7fdb0806fcb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb240b2e90 0x7fdb24139240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fdb240a9b00 0x7fdb240b7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 --2- 172.21.15.19:0/1990042712 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fdb240a9720 0x7fdb2413ca40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.859 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 -- 172.21.15.19:0/1990042712 >> 172.21.15.19:0/1990042712 conn(0x7fdb24067780 msgr2=0x7fdb240b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:26.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 -- 172.21.15.19:0/1990042712 shutdown_connections 2024-05-04T17:45:26.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 -- 172.21.15.19:0/1990042712 wait complete. 2024-05-04T17:45:26.860 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.857+0000 7fdb2b360640 1 librados: shutdown 2024-05-04T17:45:26.869 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345928 for osd.0 2024-05-04T17:45:26.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.977+0000 7f85d1c2c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-04T17:45:26.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.977+0000 7f85d1c2c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-04T17:45:26.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.977+0000 7f85d1c2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:26.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.977+0000 7f85d1c2c640 1 -- 172.21.15.19:0/2523051078 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f85cc0008d0 con 0x7f85cc072050 2024-05-04T17:45:26.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.982+0000 7f85b3fff640 1 -- 172.21.15.19:0/2523051078 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f85cc0008d0 con 0x7f85cc072050 2024-05-04T17:45:26.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.982+0000 7f85b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-04T17:45:26.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.982+0000 7f85b3fff640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:26.983 INFO:teuthology.orchestra.run.smithi019.stdout:81604378631 2024-05-04T17:45:26.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 10 librados: watch_flush enter 2024-05-04T17:45:26.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 10 librados: watch_flush exit 2024-05-04T17:45:26.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 20 client.4308.objecter shutdown clearing up homeless session... 2024-05-04T17:45:26.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 10 client.4308.objecter successfully canceled tick 2024-05-04T17:45:26.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 -- 172.21.15.19:0/2523051078 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f85b806d7c0 msgr2=0x7f85b806fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 --2- 172.21.15.19:0/2523051078 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f85b806d7c0 0x7f85b806fc80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f85cc06dff0 tx=0x7f85c0006ba0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 10 monclient: shutdown 2024-05-04T17:45:26.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:26.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 -- 172.21.15.19:0/2523051078 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f85cc072050 msgr2=0x7f85cc1401e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 --2- 172.21.15.19:0/2523051078 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f85cc072050 0x7f85cc1401e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f85bc000c00 tx=0x7f85bc0046c0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 -- 172.21.15.19:0/2523051078 shutdown_connections 2024-05-04T17:45:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 --2- 172.21.15.19:0/2523051078 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f85b806d7c0 0x7f85b806fc80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 --2- 172.21.15.19:0/2523051078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85cc1441d0 0x7f85cc13b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 --2- 172.21.15.19:0/2523051078 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f85cc0b5390 0x7f85cc13ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 --2- 172.21.15.19:0/2523051078 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f85cc072050 0x7f85cc1401e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 -- 172.21.15.19:0/2523051078 >> 172.21.15.19:0/2523051078 conn(0x7f85cc0b2cb0 msgr2=0x7f85cc1423e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 -- 172.21.15.19:0/2523051078 shutdown_connections 2024-05-04T17:45:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 -- 172.21.15.19:0/2523051078 wait complete. 2024-05-04T17:45:26.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:26.985+0000 7f85d1c2c640 1 librados: shutdown 2024-05-04T17:45:27.001 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378632 got 81604378631 for osd.6 2024-05-04T17:45:27.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.081+0000 7f650bfff640 1 -- 172.21.15.19:0/4122109576 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f651407bc50 con 0x7f65240a9fb0 2024-05-04T17:45:27.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.081+0000 7f650bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-04T17:45:27.082 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.081+0000 7f650bfff640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:27.082 INFO:teuthology.orchestra.run.smithi019.stdout:60129542150 2024-05-04T17:45:27.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f732047b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-04T17:45:27.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f732047b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-04T17:45:27.083 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f732047b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:27.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f732047b640 1 -- 172.21.15.19:0/3103994419 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f7318140950 con 0x7f73180aa890 2024-05-04T17:45:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 10 librados: watch_flush enter 2024-05-04T17:45:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 10 librados: watch_flush exit 2024-05-04T17:45:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 20 client.4393.objecter shutdown clearing up homeless session... 2024-05-04T17:45:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 10 client.4393.objecter successfully canceled tick 2024-05-04T17:45:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 -- 172.21.15.19:0/4122109576 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f651006d7c0 msgr2=0x7f651006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 --2- 172.21.15.19:0/4122109576 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f651006d7c0 0x7f651006fc80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f65240b3390 tx=0x7f6518007990 comp rx=0 tx=0).stop 2024-05-04T17:45:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 10 monclient: shutdown 2024-05-04T17:45:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:27.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 -- 172.21.15.19:0/4122109576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240a9fb0 msgr2=0x7f6524145760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 --2- 172.21.15.19:0/4122109576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240a9fb0 0x7f6524145760 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6514009990 tx=0x7f65140046c0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 -- 172.21.15.19:0/4122109576 shutdown_connections 2024-05-04T17:45:27.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 --2- 172.21.15.19:0/4122109576 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f651006d7c0 0x7f651006fc80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 --2- 172.21.15.19:0/4122109576 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f65241404e0 0x7f6524132c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 --2- 172.21.15.19:0/4122109576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f65240b27f0 0x7f652406e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 --2- 172.21.15.19:0/4122109576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65240a9fb0 0x7f6524145760 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 -- 172.21.15.19:0/4122109576 >> 172.21.15.19:0/4122109576 conn(0x7f65240b4e90 msgr2=0x7f6524136c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:27.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 -- 172.21.15.19:0/4122109576 shutdown_connections 2024-05-04T17:45:27.086 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 -- 172.21.15.19:0/4122109576 wait complete. 2024-05-04T17:45:27.087 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.083+0000 7f6529eac640 1 librados: shutdown 2024-05-04T17:45:27.095 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.2 2024-05-04T17:45:27.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.099+0000 7fa2027fc640 1 -- 172.21.15.19:0/2373148421 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa208018020 con 0x7fa20c0a9720 2024-05-04T17:45:27.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.099+0000 7fa2027fc640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa1f006d7f0 0x7fa1f006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.100+0000 7fa211440640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa1f006d7f0 0x7fa1f006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.100+0000 7fa211440640 10 monclient: get_auth_request con 0x7fa1f006d7f0 auth_method 0 2024-05-04T17:45:27.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.100+0000 7fa211440640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa1f006d7f0 0x7fa1f006fcb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0b4710 tx=0x7fa1fc008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:27.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.100+0000 7fa2027fc640 10 client.4396.objecter ms_handle_connect 0x7fa1f006d7f0 2024-05-04T17:45:27.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.106+0000 7fa2027fc640 1 -- 172.21.15.19:0/2373148421 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fa2080576d0 con 0x7fa20c0a9720 2024-05-04T17:45:27.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.106+0000 7fa2027fc640 10 client.4396.objecter ms_dispatch 0x7fa20c000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:27.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.106+0000 7fa2027fc640 3 client.4396.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:27.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.106+0000 7fa2027fc640 3 client.4396.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:27.107 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.106+0000 7fa2027fc640 20 client.4396.objecter dump_active .. 0 homeless 2024-05-04T17:45:27.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.119+0000 7fa2027fc640 1 -- 172.21.15.19:0/2373148421 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa20c13ea40 con 0x7fa20c0a9720 2024-05-04T17:45:27.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.119+0000 7fa2027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:27.120 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.119+0000 7fa2027fc640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:27.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.167+0000 7f730effd640 1 -- 172.21.15.19:0/3103994419 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f731004d030 con 0x7f73180aa890 2024-05-04T17:45:27.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.167+0000 7f730effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-04T17:45:27.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.167+0000 7f730effd640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:27.168 INFO:teuthology.orchestra.run.smithi019.stdout:90194313224 2024-05-04T17:45:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 10 librados: watch_flush enter 2024-05-04T17:45:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 10 librados: watch_flush exit 2024-05-04T17:45:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 20 client.4390.objecter shutdown clearing up homeless session... 2024-05-04T17:45:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 10 client.4390.objecter successfully canceled tick 2024-05-04T17:45:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 -- 172.21.15.19:0/3103994419 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f72fc06d7c0 msgr2=0x7f72fc06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f72fc06d7c0 0x7f72fc06fc80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7314005020 tx=0x7f7314007ef0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 10 monclient: shutdown 2024-05-04T17:45:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 -- 172.21.15.19:0/3103994419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73180aa890 msgr2=0x7f7318140ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73180aa890 0x7f7318140ca0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f73180b91d0 tx=0x7f7310004a40 comp rx=0 tx=0).stop 2024-05-04T17:45:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 -- 172.21.15.19:0/3103994419 shutdown_connections 2024-05-04T17:45:27.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f72fc06d7c0 0x7f72fc06fc80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73180aa890 0x7f7318140ca0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f73180a9b00 0x7f73181437c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 --2- 172.21.15.19:0/3103994419 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f73180a9720 0x7f7318143280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 -- 172.21.15.19:0/3103994419 >> 172.21.15.19:0/3103994419 conn(0x7f7318067780 msgr2=0x7f73180b3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:27.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 -- 172.21.15.19:0/3103994419 shutdown_connections 2024-05-04T17:45:27.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 -- 172.21.15.19:0/3103994419 wait complete. 2024-05-04T17:45:27.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.169+0000 7f732047b640 1 librados: shutdown 2024-05-04T17:45:27.181 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313224 for osd.1 2024-05-04T17:45:27.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.314+0000 7fa213ea5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-04T17:45:27.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.314+0000 7fa213ea5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-04T17:45:27.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.314+0000 7fa213ea5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:27.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.314+0000 7fa213ea5640 1 -- 172.21.15.19:0/2373148421 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fa20c06a700 con 0x7fa20c0a9720 2024-05-04T17:45:27.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.318+0000 7fa2027fc640 1 -- 172.21.15.19:0/2373148421 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa20801f020 con 0x7fa20c0a9720 2024-05-04T17:45:27.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.318+0000 7fa2027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-05-04T17:45:27.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.318+0000 7fa2027fc640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:27.319 INFO:teuthology.orchestra.run.smithi019.stdout:60129542150 2024-05-04T17:45:27.321 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 10 librados: watch_flush enter 2024-05-04T17:45:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 10 librados: watch_flush exit 2024-05-04T17:45:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 20 client.4396.objecter shutdown clearing up homeless session... 2024-05-04T17:45:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 10 client.4396.objecter successfully canceled tick 2024-05-04T17:45:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 -- 172.21.15.19:0/2373148421 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa1f006d7f0 msgr2=0x7fa1f006fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa1f006d7f0 0x7fa1f006fcb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0b4710 tx=0x7fa1fc008040 comp rx=0 tx=0).stop 2024-05-04T17:45:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 10 monclient: shutdown 2024-05-04T17:45:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 -- 172.21.15.19:0/2373148421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa20c0a9720 msgr2=0x7fa20c0ac960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa20c0a9720 0x7fa20c0ac960 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa20800b3e0 tx=0x7fa20800b8a0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 -- 172.21.15.19:0/2373148421 shutdown_connections 2024-05-04T17:45:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fa1f006d7f0 0x7fa1f006fcb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.322 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fa20c0b2e90 0x7fa20c144570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fa20c0a9b00 0x7fa20c144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 --2- 172.21.15.19:0/2373148421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa20c0a9720 0x7fa20c0ac960 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 -- 172.21.15.19:0/2373148421 >> 172.21.15.19:0/2373148421 conn(0x7fa20c067780 msgr2=0x7fa20c0b8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:27.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 -- 172.21.15.19:0/2373148421 shutdown_connections 2024-05-04T17:45:27.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 -- 172.21.15.19:0/2373148421 wait complete. 2024-05-04T17:45:27.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.320+0000 7fa213ea5640 1 librados: shutdown 2024-05-04T17:45:27.332 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.7 2024-05-04T17:45:27.332 DEBUG:teuthology.parallel:result is None 2024-05-04T17:45:27.585 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-04T17:45:27.645 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-04T17:45:27.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.689+0000 7f3d8ddd4640 10 monclient: get_monmap_and_config 2024-05-04T17:45:27.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.689+0000 7f3d8ddd4640 10 monclient: build_initial_monmap 2024-05-04T17:45:27.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.689+0000 7f3d8ddd4640 10 monclient: monmap: 2024-05-04T17:45:27.690 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:27.690 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.690 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:27.690409+0000 2024-05-04T17:45:27.690 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:27.690409+0000 2024-05-04T17:45:27.691 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:27.691 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.691 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:27.691 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:27.691 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:27.691 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:27.691 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:27.691 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.690+0000 7f3d8ddd4640 1 Processor -- start 2024-05-04T17:45:27.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.690+0000 7f3d8ddd4640 1 -- start start 2024-05-04T17:45:27.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.690+0000 7f3d8ddd4640 10 monclient: init 2024-05-04T17:45:27.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:27.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d880a9fb0 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3d881404e0 0x7f3d880aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 10 monclient(hunting): picked mon.noname-c con 0x7f3d881404e0 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f3d880b27f0 0x7f3d881463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 10 monclient(hunting): picked mon.noname-a con 0x7f3d880b27f0 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d880ab880 con 0x7f3d880a9fb0 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f3d88146930 con 0x7f3d880b27f0 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f3d88146ab0 con 0x7f3d881404e0 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d8ddd4640 10 monclient(hunting): authenticate will time out at 1349.804800s 2024-05-04T17:45:27.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d86ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3d881404e0 0x7f3d880aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d87fff640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f3d880b27f0 0x7f3d881463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d86ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3d881404e0 0x7f3d880aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36456/0 (socket says 172.21.15.19:36456) 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d86ffd640 1 -- 172.21.15.19:0/3818418662 learned_addr learned my addr 172.21.15.19:0/3818418662 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.691+0000 7f3d86ffd640 10 monclient(hunting): get_auth_request con 0x7f3d881404e0 auth_method 0 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.692+0000 7f3d86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.692+0000 7f3d86ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.692+0000 7f3d86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.692+0000 7f3d87fff640 10 monclient(hunting): get_auth_request con 0x7f3d880b27f0 auth_method 0 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.692+0000 7f3d87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.692+0000 7f3d87fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.692+0000 7f3d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.692+0000 7f3d86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.692+0000 7f3d86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.692+0000 7f3d86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.693+0000 7f3d87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.693+0000 7f3d87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:27.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.693+0000 7f3d87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:27.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.694+0000 7f3d86ffd640 10 monclient(hunting): handle_auth_done global_id 4399 payload 274 2024-05-04T17:45:27.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.694+0000 7f3d86ffd640 1 -- 172.21.15.19:0/3818418662 >> v2:172.21.15.19:3300/0 conn(0x7f3d880b27f0 msgr2=0x7f3d881463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:27.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.694+0000 7f3d86ffd640 1 --2- 172.21.15.19:0/3818418662 >> v2:172.21.15.19:3300/0 conn(0x7f3d880b27f0 0x7f3d881463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.694+0000 7f3d86ffd640 1 -- 172.21.15.19:0/3818418662 >> v1:172.21.15.19:6789/0 conn(0x7f3d880a9fb0 legacy=0x7f3d8813fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.694+0000 7f3d86ffd640 10 monclient: _finish_hunting 0 2024-05-04T17:45:27.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.694+0000 7f3d86ffd640 1 monclient: found mon.noname-c 2024-05-04T17:45:27.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.694+0000 7f3d86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:27.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.694+0000 7f3d86ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:27.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.694+0000 7f3d86ffd640 1 -- 172.21.15.19:0/3818418662 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d88146c60 con 0x7f3d881404e0 2024-05-04T17:45:27.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.696+0000 7f3d86ffd640 1 --2- 172.21.15.19:0/3818418662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d881404e0 0x7f3d880aa390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3d880ab170 tx=0x7f3d7802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f21df4b150ef262 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:27.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 1 -- 172.21.15.19:0/3818418662 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78035020 con 0x7f3d881404e0 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 10 monclient: dump: 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 10 monclient: _finish_auth 0 2024-05-04T17:45:27.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 10 monclient: _check_auth_tickets 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 1 -- 172.21.15.19:0/3818418662 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d78031870 con 0x7f3d881404e0 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 1 -- 172.21.15.19:0/3818418662 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78031c40 con 0x7f3d881404e0 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d8ddd4640 5 monclient: authenticate success, global_id 4399 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d8ddd4640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d8ddd4640 4 set_mon_vals no callback set 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d867fc640 10 monclient: dump: 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.702 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.703 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:27.703 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.703 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:27.703 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:27.703 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:27.703 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d8ddd4640 10 monclient: shutdown 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d8ddd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/3818418662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d881404e0 msgr2=0x7f3d880aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.700+0000 7f3d8ddd4640 1 --2- 172.21.15.19:0/3818418662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d881404e0 0x7f3d880aa390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3d880ab170 tx=0x7f3d7802b910 comp rx=0 tx=0).stop 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.701+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/3818418662 shutdown_connections 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.701+0000 7f3d8ddd4640 1 --2- 172.21.15.19:0/3818418662 >> v2:172.21.15.19:3300/0 conn(0x7f3d880b27f0 0x7f3d881463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.701+0000 7f3d8ddd4640 1 --2- 172.21.15.19:0/3818418662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d881404e0 0x7f3d880aa390 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.701+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/3818418662 >> 172.21.15.19:0/3818418662 conn(0x7f3d880b4e90 msgr2=0x7f3d880b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.701+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/3818418662 shutdown_connections 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.701+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/3818418662 wait complete. 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.701+0000 7f3d8ddd4640 10 monclient: build_initial_monmap 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.701+0000 7f3d8ddd4640 10 monclient: monmap: 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:27.706 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 1 librados: starting msgr at 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 1 librados: starting objecter 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 1 Processor -- start 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 1 -- start start 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 1 librados: setting wanted keys 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 1 librados: calling monclient init 2024-05-04T17:45:27.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 10 monclient: init 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d880a9fb0 0x7f3d88140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 10 monclient(hunting): picked mon.noname-a con 0x7f3d880a9fb0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3d880b27f0 0x7f3d8813bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 10 monclient(hunting): picked mon.noname-c con 0x7f3d880b27f0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d881404e0 0x7f3d8813c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 10 monclient(hunting): picked mon.noname-b con 0x7f3d881404e0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8814d7a0 con 0x7f3d880a9fb0 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8814d620 con 0x7f3d881404e0 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.702+0000 7f3d8ddd4640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f3d8814d920 con 0x7f3d880b27f0 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d8ddd4640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:27.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d8ddd4640 10 monclient(hunting): authenticate will time out at 1349.816638s 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d877fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d880a9fb0 0x7f3d88140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d87fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d881404e0 0x7f3d8813c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d86ffd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3d880b27f0 0x7f3d8813bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d87fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d881404e0 0x7f3d8813c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36468/0 (socket says 172.21.15.19:36468) 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d87fff640 1 -- 172.21.15.19:0/90103793 learned_addr learned my addr 172.21.15.19:0/90103793 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d87fff640 10 monclient(hunting): get_auth_request con 0x7f3d881404e0 auth_method 0 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d87fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d86ffd640 10 monclient(hunting): get_auth_request con 0x7f3d880b27f0 auth_method 0 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d86ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d877fe640 10 monclient(hunting): get_auth_request con 0x7f3d880a9fb0 auth_method 0 2024-05-04T17:45:27.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:27.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d877fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:27.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.703+0000 7f3d877fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:27.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.704+0000 7f3d87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:27.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.704+0000 7f3d87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:27.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.704+0000 7f3d87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:27.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.704+0000 7f3d86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:27.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.704+0000 7f3d86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:27.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.704+0000 7f3d86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:27.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.704+0000 7f3d877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:27.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.704+0000 7f3d877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:27.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.704+0000 7f3d877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:27.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.706+0000 7f3d87fff640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2024-05-04T17:45:27.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.706+0000 7f3d87fff640 1 -- 172.21.15.19:0/90103793 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3d880b27f0 msgr2=0x7f3d8813bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.706+0000 7f3d87fff640 1 --2- 172.21.15.19:0/90103793 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3d880b27f0 0x7f3d8813bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.706+0000 7f3d87fff640 1 -- 172.21.15.19:0/90103793 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d880a9fb0 msgr2=0x7f3d88140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.706+0000 7f3d87fff640 1 --2- 172.21.15.19:0/90103793 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d880a9fb0 0x7f3d88140960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.706+0000 7f3d87fff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:27.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.706+0000 7f3d87fff640 1 monclient: found mon.noname-b 2024-05-04T17:45:27.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.706+0000 7f3d87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:27.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.706+0000 7f3d87fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:27.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.706+0000 7f3d87fff640 1 -- 172.21.15.19:0/90103793 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8813a3b0 con 0x7f3d881404e0 2024-05-04T17:45:27.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.709+0000 7f3d87fff640 1 --2- 172.21.15.19:0/90103793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d881404e0 0x7f3d8813c2c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000efc0 tx=0x7f3d8000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:27.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.709+0000 7f3d7ffff640 10 client.?.objecter ms_handle_connect 0x7f3d881404e0 2024-05-04T17:45:27.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.709+0000 7f3d7ffff640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:27.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 1 -- 172.21.15.19:0/90103793 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d80018020 con 0x7f3d881404e0 2024-05-04T17:45:27.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:27.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:27.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 10 monclient: dump: 2024-05-04T17:45:27.713 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 10 monclient: _finish_auth 0 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 1 -- 172.21.15.19:0/90103793 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d80014480 con 0x7f3d881404e0 2024-05-04T17:45:27.714 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 1 -- 172.21.15.19:0/90103793 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d80014870 con 0x7f3d881404e0 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d8ddd4640 5 monclient: authenticate success, global_id 4402 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d8ddd4640 10 monclient: _renew_subs 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d8ddd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/90103793 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d8813c800 con 0x7f3d881404e0 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d857fa640 4 set_mon_vals no callback set 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d7ffff640 10 monclient: dump: 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:27.715 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:27.716 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:27.716 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d8ddd4640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:27.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d8ddd4640 10 monclient: _renew_subs 2024-05-04T17:45:27.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d8ddd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:27.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/90103793 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d88138ab0 con 0x7f3d881404e0 2024-05-04T17:45:27.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.712+0000 7f3d8ddd4640 1 librados: init done 2024-05-04T17:45:27.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.713+0000 7f3d8ddd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:27.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.713+0000 7f3d8ddd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:27.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.713+0000 7f3d8ddd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:27.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.713+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/90103793 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d88068b10 con 0x7f3d881404e0 2024-05-04T17:45:27.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.732+0000 7f3d7ffff640 1 -- 172.21.15.19:0/90103793 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3d80005d10 con 0x7f3d881404e0 2024-05-04T17:45:27.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.732+0000 7f3d7ffff640 1 --2- 172.21.15.19:0/90103793 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f3d6806d840 0x7f3d6806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.732+0000 7f3d877fe640 1 --2- 172.21.15.19:0/90103793 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f3d6806d840 0x7f3d6806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.732+0000 7f3d877fe640 10 monclient: get_auth_request con 0x7f3d6806d840 auth_method 0 2024-05-04T17:45:27.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.733+0000 7f3d877fe640 1 --2- 172.21.15.19:0/90103793 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f3d6806d840 0x7f3d6806fd00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3d70001780 tx=0x7f3d70009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:27.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.733+0000 7f3d7ffff640 10 client.4402.objecter ms_handle_connect 0x7f3d6806d840 2024-05-04T17:45:27.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.747+0000 7f62d8d84640 10 monclient: get_monmap_and_config 2024-05-04T17:45:27.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.747+0000 7f62d8d84640 10 monclient: build_initial_monmap 2024-05-04T17:45:27.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.747+0000 7f62d8d84640 10 monclient: monmap: 2024-05-04T17:45:27.748 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:27.748 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.748 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:27.748206+0000 2024-05-04T17:45:27.748 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:27.748206+0000 2024-05-04T17:45:27.748 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:27.748 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.748 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:27.748 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:27.748 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:27.749 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:27.749 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:27.749 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.747+0000 7f62d8d84640 1 Processor -- start 2024-05-04T17:45:27.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 1 -- start start 2024-05-04T17:45:27.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 10 monclient: init 2024-05-04T17:45:27.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:27.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d40a9fb0 0x7f62d40aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 10 monclient(hunting): picked mon.noname-b con 0x7f62d40a9fb0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f62d40b27f0 0x7f62d413fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 10 monclient(hunting): picked mon.noname-c con 0x7f62d40b27f0 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62d4140310 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62d40ab880 con 0x7f62d4140310 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f62d41407a0 con 0x7f62d40b27f0 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f62d4146b00 con 0x7f62d40a9fb0 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.748+0000 7f62d8d84640 10 monclient(hunting): authenticate will time out at 1349.862473s 2024-05-04T17:45:27.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.749+0000 7f62d259c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d40a9fb0 0x7f62d40aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.749+0000 7f62d2d9d640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f62d4140310 0x7f62d41463f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:42392/0 (socket says 172.21.15.19:42392) 2024-05-04T17:45:27.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.749+0000 7f62d2d9d640 1 -- 172.21.15.19:0/2923056914 learned_addr learned my addr 172.21.15.19:0/2923056914 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:27.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.749+0000 7f62d259c640 10 monclient(hunting): get_auth_request con 0x7f62d40a9fb0 auth_method 0 2024-05-04T17:45:27.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.749+0000 7f62d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:27.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.749+0000 7f62d259c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:27.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.749+0000 7f62d259c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:27.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.750+0000 7f62d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:27.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.750+0000 7f62d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:27.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.750+0000 7f62d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:27.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.751+0000 7f62d259c640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2024-05-04T17:45:27.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.751+0000 7f62d259c640 1 -- 172.21.15.19:0/2923056914 >> v2:172.21.15.186:3300/0 conn(0x7f62d40b27f0 msgr2=0x7f62d413fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:27.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.751+0000 7f62d259c640 1 --2- 172.21.15.19:0/2923056914 >> v2:172.21.15.186:3300/0 conn(0x7f62d40b27f0 0x7f62d413fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.751+0000 7f62d259c640 1 -- 172.21.15.19:0/2923056914 >> v1:172.21.15.19:6789/0 conn(0x7f62d4140310 legacy=0x7f62d41463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.751+0000 7f62d259c640 10 monclient: _finish_hunting 0 2024-05-04T17:45:27.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.751+0000 7f62d259c640 1 monclient: found mon.noname-b 2024-05-04T17:45:27.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.751+0000 7f62d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:27.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.752+0000 7f62d259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:45:27.753 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.752+0000 7f62d259c640 1 -- 172.21.15.19:0/2923056914 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d4146cb0 con 0x7f62d40a9fb0 2024-05-04T17:45:27.754 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.753+0000 7f62d259c640 1 --2- 172.21.15.19:0/2923056914 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d40a9fb0 0x7f62d40aa390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0097a0 tx=0x7f62cc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb82ce6b23737b1b server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:27.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 1 -- 172.21.15.19:0/2923056914 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc0315a0 con 0x7f62d40a9fb0 2024-05-04T17:45:27.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:27.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:27.758 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 10 monclient: dump: 2024-05-04T17:45:27.758 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:27.758 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 10 monclient: _finish_auth 0 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 10 monclient: _check_auth_tickets 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d8d84640 5 monclient: authenticate success, global_id 4298 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 1 -- 172.21.15.19:0/2923056914 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62cc037070 con 0x7f62d40a9fb0 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d8d84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:27.759 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d8d84640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d8d84640 4 set_mon_vals no callback set 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 1 -- 172.21.15.19:0/2923056914 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc031960 con 0x7f62d40a9fb0 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.757+0000 7f62d159a640 10 monclient: dump: 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:27.760 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.758+0000 7f62d8d84640 10 monclient: shutdown 2024-05-04T17:45:27.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.758+0000 7f62d8d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:27.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.758+0000 7f62d8d84640 1 -- 172.21.15.19:0/2923056914 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d40a9fb0 msgr2=0x7f62d40aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.758+0000 7f62d8d84640 1 --2- 172.21.15.19:0/2923056914 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d40a9fb0 0x7f62d40aa390 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0097a0 tx=0x7f62cc02b840 comp rx=0 tx=0).stop 2024-05-04T17:45:27.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.758+0000 7f62d8d84640 1 -- 172.21.15.19:0/2923056914 shutdown_connections 2024-05-04T17:45:27.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.758+0000 7f62d8d84640 1 --2- 172.21.15.19:0/2923056914 >> v2:172.21.15.186:3300/0 conn(0x7f62d40b27f0 0x7f62d413fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.758+0000 7f62d8d84640 1 --2- 172.21.15.19:0/2923056914 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d40a9fb0 0x7f62d40aa390 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.758+0000 7f62d8d84640 1 -- 172.21.15.19:0/2923056914 >> 172.21.15.19:0/2923056914 conn(0x7f62d40b4e90 msgr2=0x7f62d40b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:27.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.758+0000 7f62d8d84640 1 -- 172.21.15.19:0/2923056914 shutdown_connections 2024-05-04T17:45:27.763 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.758+0000 7f62d8d84640 1 -- 172.21.15.19:0/2923056914 wait complete. 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.759+0000 7f62d8d84640 10 monclient: build_initial_monmap 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.759+0000 7f62d8d84640 10 monclient: monmap: 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.759+0000 7f62d8d84640 1 librados: starting msgr at 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.759+0000 7f62d8d84640 1 librados: starting objecter 2024-05-04T17:45:27.764 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 1 Processor -- start 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 1 -- start start 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 1 librados: setting wanted keys 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 1 librados: calling monclient init 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 10 monclient: init 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62d40a9fb0 0x7f62d4143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 10 monclient(hunting): picked mon.noname-b con 0x7f62d40a9fb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62d40b27f0 0x7f62d41444e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 10 monclient(hunting): picked mon.noname-a con 0x7f62d40b27f0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d4140310 0x7f62d41416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 10 monclient(hunting): picked mon.noname-c con 0x7f62d4140310 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.765 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f62d414d7a0 con 0x7f62d40b27f0 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f62d414d620 con 0x7f62d40a9fb0 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f62d414d920 con 0x7f62d4140310 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.760+0000 7f62d8d84640 10 monclient(hunting): authenticate will time out at 1349.874298s 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d2d9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d4140310 0x7f62d41416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d1d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62d40b27f0 0x7f62d41444e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d2d9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d4140310 0x7f62d41416b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:37148/0 (socket says 172.21.15.19:37148) 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d2d9d640 1 -- 172.21.15.19:0/584156358 learned_addr learned my addr 172.21.15.19:0/584156358 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d2d9d640 10 monclient(hunting): get_auth_request con 0x7f62d4140310 auth_method 0 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d2d9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:27.766 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d1d9b640 10 monclient(hunting): get_auth_request con 0x7f62d40b27f0 auth_method 0 2024-05-04T17:45:27.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:27.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d1d9b640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:27.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:27.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:27.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:27.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.761+0000 7f62d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:27.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.762+0000 7f62d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:27.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.762+0000 7f62d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:27.767 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.762+0000 7f62d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:27.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.764+0000 7f62d2d9d640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-05-04T17:45:27.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.764+0000 7f62d2d9d640 1 -- 172.21.15.19:0/584156358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62d40a9fb0 msgr2=0x7f62d4143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.764+0000 7f62d2d9d640 1 --2- 172.21.15.19:0/584156358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62d40a9fb0 0x7f62d4143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.764+0000 7f62d2d9d640 1 -- 172.21.15.19:0/584156358 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62d40b27f0 msgr2=0x7f62d41444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.764+0000 7f62d2d9d640 1 --2- 172.21.15.19:0/584156358 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62d40b27f0 0x7f62d41444e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.764+0000 7f62d2d9d640 10 monclient: _finish_hunting 0 2024-05-04T17:45:27.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.764+0000 7f62d2d9d640 1 monclient: found mon.noname-c 2024-05-04T17:45:27.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.764+0000 7f62d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:27.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.764+0000 7f62d2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T17:45:27.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.764+0000 7f62d2d9d640 1 -- 172.21.15.19:0/584156358 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d4142370 con 0x7f62d4140310 2024-05-04T17:45:27.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.767+0000 7f62d2d9d640 1 --2- 172.21.15.19:0/584156358 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d4140310 0x7f62d41416b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f62c400cbc0 tx=0x7f62c40021c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:27.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.767+0000 7f62c2ffd640 10 client.?.objecter ms_handle_connect 0x7f62d4140310 2024-05-04T17:45:27.769 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.767+0000 7f62c2ffd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:27.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 1 -- 172.21.15.19:0/584156358 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4017070 con 0x7f62d4140310 2024-05-04T17:45:27.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:27.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:27.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 10 monclient: dump: 2024-05-04T17:45:27.772 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:27.772 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.772 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.772 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.772 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:27.772 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.772 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:27.772 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 10 monclient: _finish_auth 0 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 10 monclient: _check_auth_tickets 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 1 -- 172.21.15.19:0/584156358 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62c400ac00 con 0x7f62d4140310 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 1 -- 172.21.15.19:0/584156358 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4012400 con 0x7f62d4140310 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c2ffd640 10 monclient: dump: 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.773 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62d8d84640 5 monclient: authenticate success, global_id 4301 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62d8d84640 10 monclient: _renew_subs 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.771+0000 7f62d8d84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.771+0000 7f62d8d84640 1 -- 172.21.15.19:0/584156358 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d4142950 con 0x7f62d4140310 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.770+0000 7f62c3fff640 4 set_mon_vals no callback set 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.771+0000 7f62d8d84640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.771+0000 7f62d8d84640 10 monclient: _renew_subs 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.771+0000 7f62d8d84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:27.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.771+0000 7f62d8d84640 1 -- 172.21.15.19:0/584156358 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d4141bf0 con 0x7f62d4140310 2024-05-04T17:45:27.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.771+0000 7f62d8d84640 1 librados: init done 2024-05-04T17:45:27.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.772+0000 7f62d8d84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:27.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.772+0000 7f62d8d84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:27.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.772+0000 7f62d8d84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:27.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.772+0000 7f62d8d84640 1 -- 172.21.15.19:0/584156358 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d4068b10 con 0x7f62d4140310 2024-05-04T17:45:27.813 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-04T17:45:27.870 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-04T17:45:27.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.914+0000 7fe4215bd640 10 monclient: get_monmap_and_config 2024-05-04T17:45:27.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.914+0000 7fe4215bd640 10 monclient: build_initial_monmap 2024-05-04T17:45:27.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.914+0000 7fe4215bd640 10 monclient: monmap: 2024-05-04T17:45:27.915 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:27.915 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.915 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:27.915389+0000 2024-05-04T17:45:27.915 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:27.915389+0000 2024-05-04T17:45:27.916 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:27.916 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.916 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:27.916 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:27.916 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:27.916 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:27.916 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:27.916 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.915+0000 7fe4215bd640 1 Processor -- start 2024-05-04T17:45:27.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.915+0000 7fe4215bd640 1 -- start start 2024-05-04T17:45:27.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.915+0000 7fe4215bd640 10 monclient: init 2024-05-04T17:45:27.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.915+0000 7fe4215bd640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:27.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.915+0000 7fe4215bd640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-05-04T17:45:27.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.915+0000 7fe4215bd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe41c0a9720 0x7fe41c0aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.915+0000 7fe4215bd640 10 monclient(hunting): picked mon.noname-c con 0x7fe41c0a9720 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:27.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.916+0000 7fe4215bd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe41c0a9b00 0x7fe41c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.916+0000 7fe4215bd640 10 monclient(hunting): picked mon.noname-b con 0x7fe41c0a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:27.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.916+0000 7fe4215bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe41c146890 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:27.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.916+0000 7fe4215bd640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe41c0abc40 con 0x7fe41c146890 2024-05-04T17:45:27.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.916+0000 7fe4215bd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.916+0000 7fe4215bd640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fe41c146090 con 0x7fe41c0a9720 2024-05-04T17:45:27.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.916+0000 7fe4215bd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.916+0000 7fe4215bd640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fe41c146370 con 0x7fe41c0a9b00 2024-05-04T17:45:27.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.916+0000 7fe4215bd640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:27.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.916+0000 7fe4215bd640 10 monclient(hunting): authenticate will time out at 1350.029712s 2024-05-04T17:45:27.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.965+0000 7f3d7ffff640 1 -- 172.21.15.19:0/90103793 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f3d80058570 con 0x7f3d881404e0 2024-05-04T17:45:27.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.965+0000 7f3d7ffff640 10 client.4402.objecter ms_dispatch 0x7f3d88000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:27.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.965+0000 7f3d7ffff640 3 client.4402.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:27.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.965+0000 7f3d7ffff640 3 client.4402.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:27.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.966+0000 7f3d7ffff640 20 client.4402.objecter dump_active .. 0 homeless 2024-05-04T17:45:27.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.971+0000 7f5f1d54b640 10 monclient: get_monmap_and_config 2024-05-04T17:45:27.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.971+0000 7f5f1d54b640 10 monclient: build_initial_monmap 2024-05-04T17:45:27.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.971+0000 7f5f1d54b640 10 monclient: monmap: 2024-05-04T17:45:27.972 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:27.972 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.972 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:27.972376+0000 2024-05-04T17:45:27.972 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:27.972376+0000 2024-05-04T17:45:27.972 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:27.972 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.973 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:27.973 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:27.973 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:27.973 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:27.973 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:27.973 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 1 Processor -- start 2024-05-04T17:45:27.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 1 -- start start 2024-05-04T17:45:27.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 10 monclient: init 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f180a9720 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f5f180a9b00 0x7f5f180aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 10 monclient(hunting): picked mon.noname-a con 0x7f5f180a9b00 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5f180b2e90 0x7f5f1813fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 10 monclient(hunting): picked mon.noname-b con 0x7f5f180b2e90 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f180abc40 con 0x7f5f180a9720 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f5f18146fa0 con 0x7f5f180a9b00 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f5f18146090 con 0x7f5f180b2e90 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.972+0000 7f5f1d54b640 10 monclient(hunting): authenticate will time out at 1350.086614s 2024-05-04T17:45:27.975 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.973+0000 7f5f0ffff640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f5f180a9b00 0x7f5f180aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.974+0000 7f5f16ffd640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f5f180a9720 0x7f5f18146890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:42402/0 (socket says 172.21.15.19:42402) 2024-05-04T17:45:27.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.974+0000 7f5f16ffd640 1 -- 172.21.15.19:0/1464620189 learned_addr learned my addr 172.21.15.19:0/1464620189 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:27.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.974+0000 7f5f0ffff640 10 monclient(hunting): get_auth_request con 0x7f5f180a9b00 auth_method 0 2024-05-04T17:45:27.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.974+0000 7f5f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:27.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.974+0000 7f5f0ffff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:27.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.974+0000 7f5f0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:27.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.975+0000 7f5f0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:27.976 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.975+0000 7f5f0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:27.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.975+0000 7f5f0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:27.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.976+0000 7f3d7ffff640 1 -- 172.21.15.19:0/90103793 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3d80022030 con 0x7f3d881404e0 2024-05-04T17:45:27.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.976+0000 7f3d7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:27.977 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.976+0000 7f3d7ffff640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:27.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.977+0000 7f5f0ffff640 10 monclient(hunting): handle_auth_done global_id 4323 payload 274 2024-05-04T17:45:27.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.977+0000 7f5f0ffff640 1 -- 172.21.15.19:0/1464620189 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5f180b2e90 msgr2=0x7f5f1813fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:27.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.977+0000 7f5f0ffff640 1 --2- 172.21.15.19:0/1464620189 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5f180b2e90 0x7f5f1813fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.977+0000 7f5f0ffff640 1 -- 172.21.15.19:0/1464620189 >> v1:172.21.15.19:6789/0 conn(0x7f5f180a9720 legacy=0x7f5f18146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.977+0000 7f5f0ffff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:27.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.977+0000 7f5f0ffff640 1 monclient: found mon.noname-a 2024-05-04T17:45:27.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.977+0000 7f5f0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:27.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.977+0000 7f5f0ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:27.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.977+0000 7f5f0ffff640 1 -- 172.21.15.19:0/1464620189 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f18146370 con 0x7f5f180a9b00 2024-05-04T17:45:27.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.980+0000 7fe41b7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe41c146890 0x7fe41c13fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:46524/0 (socket says 172.21.15.19:46524) 2024-05-04T17:45:27.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.980+0000 7fe41b7fe640 1 -- 172.21.15.19:0/3377520636 learned_addr learned my addr 172.21.15.19:0/3377520636 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:27.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.980+0000 7fe41affd640 1 --2- 172.21.15.19:0/3377520636 >> v2:172.21.15.186:3300/0 conn(0x7fe41c0a9720 0x7fe41c0aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.981+0000 7fe41affd640 10 monclient(hunting): get_auth_request con 0x7fe41c0a9720 auth_method 0 2024-05-04T17:45:27.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.981+0000 7fe41affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:27.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.981+0000 7fe41affd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:27.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.981+0000 7fe41affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:27.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.982+0000 7fe41affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:27.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.982+0000 7fe41affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:27.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.982+0000 7fe41affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:27.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.983+0000 7f5f0ffff640 1 --2- 172.21.15.19:0/1464620189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5f180a9b00 0x7f5f180aa890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5f000097a0 tx=0x7f5f0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3c14ac07f4bb94 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:27.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.983+0000 7fe41a7fc640 1 -- 172.21.15.19:0/3377520636 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1819675287 0 0) 0x7fe41c0abc40 con 0x7fe41c146890 2024-05-04T17:45:27.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.983+0000 7fe41a7fc640 10 monclient(hunting): _negotiate 2024-05-04T17:45:27.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.983+0000 7fe41a7fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:27.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.983+0000 7fe41a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:27.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.983+0000 7fe41a7fc640 10 monclient(hunting): my global_id is 4408 2024-05-04T17:45:27.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.983+0000 7fe41a7fc640 1 -- 172.21.15.19:0/3377520636 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe408003520 con 0x7fe41c146890 2024-05-04T17:45:27.985 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.985+0000 7fe41affd640 10 monclient(hunting): handle_auth_done global_id 4405 payload 274 2024-05-04T17:45:27.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.985+0000 7fe41affd640 1 -- 172.21.15.19:0/3377520636 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe41c0a9b00 msgr2=0x7fe41c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:27.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.985+0000 7fe41affd640 1 --2- 172.21.15.19:0/3377520636 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe41c0a9b00 0x7fe41c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.985+0000 7fe41affd640 1 -- 172.21.15.19:0/3377520636 >> v1:172.21.15.186:6789/0 conn(0x7fe41c146890 legacy=0x7fe41c13fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.985+0000 7fe41affd640 10 monclient: _finish_hunting 0 2024-05-04T17:45:27.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.985+0000 7fe41affd640 1 monclient: found mon.noname-c 2024-05-04T17:45:27.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.985+0000 7fe41affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:27.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.985+0000 7fe41affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:27.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.985+0000 7fe41affd640 1 -- 172.21.15.19:0/3377520636 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe41c1405b0 con 0x7fe41c0a9720 2024-05-04T17:45:27.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 1 -- 172.21.15.19:0/1464620189 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f00035020 con 0x7f5f180a9b00 2024-05-04T17:45:27.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:27.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:27.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 10 monclient: dump: 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 10 monclient: _finish_auth 0 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 10 monclient: _check_auth_tickets 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:27.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 1 -- 172.21.15.19:0/1464620189 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f00031830 con 0x7f5f180a9b00 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f1d54b640 5 monclient: authenticate success, global_id 4323 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 1 -- 172.21.15.19:0/1464620189 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f00031c00 con 0x7f5f180a9b00 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f167fc640 10 monclient: dump: 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:27.990 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:27.991 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f1d54b640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:27.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.987+0000 7f5f1d54b640 4 set_mon_vals no callback set 2024-05-04T17:45:27.991 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.988+0000 7fe41affd640 1 --2- 172.21.15.19:0/3377520636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe41c0a9720 0x7fe41c0aa890 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe40c0097a0 tx=0x7fe40c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68db370b7c8a7fa0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:27.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.988+0000 7f5f1d54b640 10 monclient: shutdown 2024-05-04T17:45:27.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.988+0000 7f5f1d54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:27.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.988+0000 7f5f1d54b640 1 -- 172.21.15.19:0/1464620189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5f180a9b00 msgr2=0x7f5f180aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:27.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.988+0000 7f5f1d54b640 1 --2- 172.21.15.19:0/1464620189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5f180a9b00 0x7f5f180aa890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5f000097a0 tx=0x7f5f0002b8d0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.988+0000 7f5f1d54b640 1 -- 172.21.15.19:0/1464620189 shutdown_connections 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.988+0000 7f5f1d54b640 1 --2- 172.21.15.19:0/1464620189 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5f180b2e90 0x7f5f1813fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.988+0000 7f5f1d54b640 1 --2- 172.21.15.19:0/1464620189 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5f180a9b00 0x7f5f180aa890 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.988+0000 7f5f1d54b640 1 -- 172.21.15.19:0/1464620189 >> 172.21.15.19:0/1464620189 conn(0x7f5f18067780 msgr2=0x7f5f180b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.988+0000 7f5f1d54b640 1 -- 172.21.15.19:0/1464620189 shutdown_connections 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.988+0000 7f5f1d54b640 1 -- 172.21.15.19:0/1464620189 wait complete. 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.989+0000 7f5f1d54b640 10 monclient: build_initial_monmap 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.989+0000 7f5f1d54b640 10 monclient: monmap: 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.994 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.990+0000 7f5f1d54b640 1 librados: starting msgr at 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.990+0000 7f5f1d54b640 1 librados: starting objecter 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.990+0000 7f5f1d54b640 1 Processor -- start 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.990+0000 7f5f1d54b640 1 -- start start 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.990+0000 7f5f1d54b640 1 librados: setting wanted keys 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.990+0000 7f5f1d54b640 1 librados: calling monclient init 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.990+0000 7f5f1d54b640 10 monclient: init 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5f180a9720 0x7f5f18147a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 10 monclient(hunting): picked mon.noname-a con 0x7f5f180a9720 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5f180a9b00 0x7f5f1814c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 10 monclient(hunting): picked mon.noname-c con 0x7f5f180a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f180b2e90 0x7f5f1814cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 10 monclient(hunting): picked mon.noname-b con 0x7f5f180b2e90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5f181407b0 con 0x7f5f180a9720 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5f18140630 con 0x7f5f180b2e90 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f5f1814d420 con 0x7f5f180a9b00 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.991+0000 7f5f1d54b640 10 monclient(hunting): authenticate will time out at 1350.105058s 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7f5f16ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5f180a9720 0x7f5f18147a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7f5f16ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5f180a9720 0x7f5f18147a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37426/0 (socket says 172.21.15.19:37426) 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7f5f16ffd640 1 -- 172.21.15.19:0/529660183 learned_addr learned my addr 172.21.15.19:0/529660183 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7f5f16ffd640 10 monclient(hunting): get_auth_request con 0x7f5f180a9720 auth_method 0 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7f5f16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:27.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7f5f16ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:27.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7f5f16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:27.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 1 -- 172.21.15.19:0/3377520636 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40c035020 con 0x7fe41c0a9720 2024-05-04T17:45:27.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:27.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7f5f16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:27.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7f5f16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:27.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7f5f16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 10 monclient: dump: 2024-05-04T17:45:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:27.998 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:27.998 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 10 monclient: _finish_auth 0 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 10 monclient: _check_auth_tickets 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 1 -- 172.21.15.19:0/3377520636 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe40c031830 con 0x7fe41c0a9720 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe4215bd640 5 monclient: authenticate success, global_id 4405 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe4215bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 1 -- 172.21.15.19:0/3377520636 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40c031c00 con 0x7fe41c0a9720 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe41a7fc640 10 monclient: dump: 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:27.999 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.000 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.000 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.000 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.000 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe4215bd640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:28.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.992+0000 7fe4215bd640 4 set_mon_vals no callback set 2024-05-04T17:45:28.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7f5f177fe640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f180b2e90 0x7f5f1814cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7f5f177fe640 10 monclient(hunting): get_auth_request con 0x7f5f180b2e90 auth_method 0 2024-05-04T17:45:28.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7f5f177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:28.000 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7f5f177fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:28.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7f5f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:28.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7fe4215bd640 10 monclient: shutdown 2024-05-04T17:45:28.001 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7fe4215bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:28.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7fe4215bd640 1 -- 172.21.15.19:0/3377520636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe41c0a9720 msgr2=0x7fe41c0aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7fe4215bd640 1 --2- 172.21.15.19:0/3377520636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe41c0a9720 0x7fe41c0aa890 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe40c0097a0 tx=0x7fe40c02b8d0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7fe4215bd640 1 -- 172.21.15.19:0/3377520636 shutdown_connections 2024-05-04T17:45:28.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7fe4215bd640 1 --2- 172.21.15.19:0/3377520636 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe41c0a9b00 0x7fe41c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7fe4215bd640 1 --2- 172.21.15.19:0/3377520636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe41c0a9720 0x7fe41c0aa890 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.993+0000 7fe4215bd640 1 -- 172.21.15.19:0/3377520636 >> 172.21.15.19:0/3377520636 conn(0x7fe41c067780 msgr2=0x7fe41c0b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:28.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.994+0000 7fe4215bd640 1 -- 172.21.15.19:0/3377520636 shutdown_connections 2024-05-04T17:45:28.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.994+0000 7fe4215bd640 1 -- 172.21.15.19:0/3377520636 wait complete. 2024-05-04T17:45:28.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.994+0000 7f5f177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:28.002 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.994+0000 7f5f177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:28.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.994+0000 7f5f177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:28.003 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-04T17:45:28.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.995+0000 7fe4215bd640 10 monclient: build_initial_monmap 2024-05-04T17:45:28.006 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.995+0000 7fe4215bd640 10 monclient: monmap: 2024-05-04T17:45:28.006 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:28.006 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:28.006 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:28.006 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:28.006 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:28.006 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.006 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:28.007 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:28.007 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:28.007 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.995+0000 7fe4215bd640 1 librados: starting msgr at 2024-05-04T17:45:28.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.995+0000 7fe4215bd640 1 librados: starting objecter 2024-05-04T17:45:28.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.995+0000 7fe4215bd640 1 Processor -- start 2024-05-04T17:45:28.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.995+0000 7fe4215bd640 1 -- start start 2024-05-04T17:45:28.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.995+0000 7fe4215bd640 1 librados: setting wanted keys 2024-05-04T17:45:28.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.995+0000 7fe4215bd640 1 librados: calling monclient init 2024-05-04T17:45:28.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.995+0000 7fe4215bd640 10 monclient: init 2024-05-04T17:45:28.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7f5f16ffd640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-05-04T17:45:28.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7f5f16ffd640 1 -- 172.21.15.19:0/529660183 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5f180a9b00 msgr2=0x7f5f1814c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7f5f16ffd640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5f180a9b00 0x7f5f1814c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7f5f16ffd640 1 -- 172.21.15.19:0/529660183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f180b2e90 msgr2=0x7f5f1814cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7f5f16ffd640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f180b2e90 0x7f5f1814cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7f5f16ffd640 10 monclient: _finish_hunting 0 2024-05-04T17:45:28.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7f5f16ffd640 1 monclient: found mon.noname-a 2024-05-04T17:45:28.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7f5f16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:28.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7f5f16ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:28.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7f5f16ffd640 1 -- 172.21.15.19:0/529660183 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f18149b80 con 0x7f5f180a9720 2024-05-04T17:45:28.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:28.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:45:28.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe41c0a9720 0x7fe41c14a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 10 monclient(hunting): picked mon.noname-a con 0x7fe41c0a9720 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe41c0a9b00 0x7fe41c139200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 10 monclient(hunting): picked mon.noname-c con 0x7fe41c0a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe41c146890 0x7fe41c139740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 10 monclient(hunting): picked mon.noname-b con 0x7fe41c146890 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c14d420 con 0x7fe41c0a9720 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe41c146f70 con 0x7fe41c146890 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fe41c14d5a0 con 0x7fe41c0a9b00 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.996+0000 7fe4215bd640 10 monclient(hunting): authenticate will time out at 1350.110320s 2024-05-04T17:45:28.010 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:27.999+0000 7fe41affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe41c0a9720 0x7fe41c14a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.000+0000 7f5f16ffd640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5f180a9720 0x7f5f18147a70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5f0800c3f0 tx=0x7f5f0800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.000+0000 7f5f0f7fe640 10 client.?.objecter ms_handle_connect 0x7f5f180a9720 2024-05-04T17:45:28.011 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.000+0000 7f5f0f7fe640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:28.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.000+0000 7fe41affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe41c0a9720 0x7fe41c14a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37434/0 (socket says 172.21.15.19:37434) 2024-05-04T17:45:28.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.000+0000 7fe41affd640 1 -- 172.21.15.19:0/3879813685 learned_addr learned my addr 172.21.15.19:0/3879813685 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:28.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.000+0000 7fe41affd640 10 monclient(hunting): get_auth_request con 0x7fe41c0a9720 auth_method 0 2024-05-04T17:45:28.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.000+0000 7fe41affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:28.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.000+0000 7fe41affd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:28.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.000+0000 7fe41affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:28.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.000+0000 7fe41b7fe640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe41c146890 0x7fe41c139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.001+0000 7fe41affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:28.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.001+0000 7fe41affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:28.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.001+0000 7fe41affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.001+0000 7fe41b7fe640 10 monclient(hunting): get_auth_request con 0x7fe41c146890 auth_method 0 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.001+0000 7fe41b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.001+0000 7fe41b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.001+0000 7fe41b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.001+0000 7fe41b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.001+0000 7fe41b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.001+0000 7fe41b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.004+0000 7fe41b7fe640 10 monclient(hunting): handle_auth_done global_id 4414 payload 824 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.004+0000 7fe41b7fe640 1 -- 172.21.15.19:0/3879813685 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe41c0a9b00 msgr2=0x7fe41c139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.004+0000 7fe41b7fe640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe41c0a9b00 0x7fe41c139200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.004+0000 7fe41b7fe640 1 -- 172.21.15.19:0/3879813685 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe41c0a9720 msgr2=0x7fe41c14a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.004+0000 7fe41b7fe640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe41c0a9720 0x7fe41c14a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.004+0000 7fe41b7fe640 10 monclient: _finish_hunting 0 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.004+0000 7fe41b7fe640 1 monclient: found mon.noname-b 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.004+0000 7fe41b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:28.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.004+0000 7fe41b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.004+0000 7fe41b7fe640 1 -- 172.21.15.19:0/3879813685 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe41c149700 con 0x7fe41c146890 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.006+0000 7fe41b7fe640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe41c146890 0x7fe41c139740 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe41400caa0 tx=0x7fe41400cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.006+0000 7fe4137fe640 10 client.?.objecter ms_handle_connect 0x7fe41c146890 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.006+0000 7fe4137fe640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 1 -- 172.21.15.19:0/3879813685 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414017070 con 0x7fe41c146890 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 10 monclient: dump: 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.016 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 1 -- 172.21.15.19:0/3879813685 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe41400ab40 con 0x7fe41c146890 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4215bd640 5 monclient: authenticate success, global_id 4414 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4215bd640 10 monclient: _renew_subs 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4215bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4215bd640 1 -- 172.21.15.19:0/3879813685 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe41c1499e0 con 0x7fe41c146890 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 1 -- 172.21.15.19:0/3879813685 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414012400 con 0x7fe41c146890 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:28.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4137fe640 10 monclient: dump: 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4215bd640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4215bd640 10 monclient: _renew_subs 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe419ffb640 4 set_mon_vals no callback set 2024-05-04T17:45:28.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4215bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:28.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4215bd640 1 -- 172.21.15.19:0/3879813685 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe41c139c80 con 0x7fe41c146890 2024-05-04T17:45:28.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.010+0000 7fe4215bd640 1 librados: init done 2024-05-04T17:45:28.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7fe4215bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7fe4215bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7fe4215bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:28.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7fe4215bd640 1 -- 172.21.15.19:0/3879813685 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe41c06a700 con 0x7fe41c146890 2024-05-04T17:45:28.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 1 -- 172.21.15.19:0/529660183 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f08017070 con 0x7f5f180a9720 2024-05-04T17:45:28.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:28.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 10 monclient: dump: 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 1 -- 172.21.15.19:0/529660183 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f08005ce0 con 0x7f5f180a9720 2024-05-04T17:45:28.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f1d54b640 5 monclient: authenticate success, global_id 4329 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f1d54b640 10 monclient: _renew_subs 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f1d54b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f1d54b640 1 -- 172.21.15.19:0/529660183 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f18149e60 con 0x7f5f180a9720 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 1 -- 172.21.15.19:0/529660183 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f080072f0 con 0x7f5f180a9720 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f0f7fe640 10 monclient: dump: 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.022 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f1d54b640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f1d54b640 10 monclient: _renew_subs 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f15ffb640 4 set_mon_vals no callback set 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f1d54b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f1d54b640 1 -- 172.21.15.19:0/529660183 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f18147fb0 con 0x7f5f180a9720 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.011+0000 7f5f1d54b640 1 librados: init done 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.012+0000 7f5f1d54b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.012+0000 7f5f1d54b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.012+0000 7f5f1d54b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:28.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.012+0000 7f5f1d54b640 1 -- 172.21.15.19:0/529660183 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f1806a700 con 0x7f5f180a9720 2024-05-04T17:45:28.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.030+0000 7fe4137fe640 1 -- 172.21.15.19:0/3879813685 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe41401a070 con 0x7fe41c146890 2024-05-04T17:45:28.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.031+0000 7fe4137fe640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe3fc06d7c0 0x7fe3fc06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.031+0000 7fe41affd640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe3fc06d7c0 0x7fe3fc06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.031+0000 7fe41affd640 10 monclient: get_auth_request con 0x7fe3fc06d7c0 auth_method 0 2024-05-04T17:45:28.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.031+0000 7fe4137fe640 10 client.4414.objecter ms_handle_connect 0x7fe3fc06d7c0 2024-05-04T17:45:28.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.031+0000 7fe41affd640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe3fc06d7c0 0x7fe3fc06fc80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe40c009560 tx=0x7fe40c008030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.032+0000 7fe4137fe640 1 -- 172.21.15.19:0/3879813685 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fe414057310 con 0x7fe41c146890 2024-05-04T17:45:28.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.032+0000 7fe4137fe640 10 client.4414.objecter ms_dispatch 0x7fe41c000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:28.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.032+0000 7fe4137fe640 3 client.4414.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:28.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.032+0000 7fe4137fe640 3 client.4414.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:28.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.032+0000 7fe4137fe640 20 client.4414.objecter dump_active .. 0 homeless 2024-05-04T17:45:28.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.034+0000 7f5f0f7fe640 1 -- 172.21.15.19:0/529660183 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5f0801a070 con 0x7f5f180a9720 2024-05-04T17:45:28.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.034+0000 7f5f0f7fe640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5ef806d840 0x7f5ef806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.034+0000 7f5f0ffff640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5ef806d840 0x7f5ef806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.034+0000 7f5f0ffff640 10 monclient: get_auth_request con 0x7f5ef806d840 auth_method 0 2024-05-04T17:45:28.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.035+0000 7f5f0f7fe640 10 client.4329.objecter ms_handle_connect 0x7f5ef806d840 2024-05-04T17:45:28.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.035+0000 7f5f0ffff640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5ef806d840 0x7f5ef806fd00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5f00036000 tx=0x7f5f00030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.039+0000 7f5f0f7fe640 1 -- 172.21.15.19:0/529660183 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f5f08056990 con 0x7f5f180a9720 2024-05-04T17:45:28.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.039+0000 7f5f0f7fe640 10 client.4329.objecter ms_dispatch 0x7f5f18000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:28.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.039+0000 7f5f0f7fe640 3 client.4329.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:28.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.039+0000 7f5f0f7fe640 3 client.4329.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:28.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.039+0000 7f5f0f7fe640 20 client.4329.objecter dump_active .. 0 homeless 2024-05-04T17:45:28.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.062+0000 7f62c2ffd640 1 -- 172.21.15.19:0/584156358 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f62c401a070 con 0x7f62d4140310 2024-05-04T17:45:28.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.062+0000 7f62c2ffd640 1 --2- 172.21.15.19:0/584156358 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f62b406d810 0x7f62b406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.062+0000 7f62d259c640 1 --2- 172.21.15.19:0/584156358 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f62b406d810 0x7f62b406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.063+0000 7f62d259c640 10 monclient: get_auth_request con 0x7f62b406d810 auth_method 0 2024-05-04T17:45:28.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.063+0000 7f62c2ffd640 10 client.4301.objecter ms_handle_connect 0x7f62b406d810 2024-05-04T17:45:28.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.063+0000 7f62d259c640 1 --2- 172.21.15.19:0/584156358 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f62b406d810 0x7f62b406fcd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f62cc036000 tx=0x7f62cc030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.078+0000 7f62c2ffd640 1 -- 172.21.15.19:0/584156358 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f62c4057390 con 0x7f62d4140310 2024-05-04T17:45:28.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.078+0000 7f62c2ffd640 10 client.4301.objecter ms_dispatch 0x7f62d4000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:28.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.078+0000 7f62c2ffd640 3 client.4301.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:28.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.078+0000 7f62c2ffd640 3 client.4301.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:28.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.078+0000 7f62c2ffd640 20 client.4301.objecter dump_active .. 0 homeless 2024-05-04T17:45:28.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.078+0000 7f62c2ffd640 1 -- 172.21.15.19:0/584156358 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f62c40592f0 con 0x7f62d4140310 2024-05-04T17:45:28.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.078+0000 7f62c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.079 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.078+0000 7f62c2ffd640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:28.096 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-04T17:45:28.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.139+0000 7f1ee483f640 10 monclient: get_monmap_and_config 2024-05-04T17:45:28.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.139+0000 7f1ee483f640 10 monclient: build_initial_monmap 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.139+0000 7f1ee483f640 10 monclient: monmap: 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:28.140752+0000 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:28.140752+0000 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:28.141 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.145+0000 7f1ee483f640 1 Processor -- start 2024-05-04T17:45:28.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.145+0000 7f1ee483f640 1 -- start start 2024-05-04T17:45:28.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.145+0000 7f1ee483f640 10 monclient: init 2024-05-04T17:45:28.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.146+0000 7f1ee483f640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:28.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.146+0000 7f1ee483f640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-05-04T17:45:28.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.146+0000 7f1ee483f640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 0x7f1ee00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.146+0000 7f1ee483f640 10 monclient(hunting): picked mon.noname-b con 0x7f1ee00b1f30 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:28.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.146+0000 7f1ee483f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ee00b33d0 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:28.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.146+0000 7f1ee483f640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f1ee01449c0 0x7f1ee0151880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.146+0000 7f1ee483f640 10 monclient(hunting): picked mon.noname-a con 0x7f1ee01449c0 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.146+0000 7f1ee483f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ee006f600 con 0x7f1ee00b33d0 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.147+0000 7f1ee483f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.147+0000 7f1ee483f640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f1ee00b2310 con 0x7f1ee01449c0 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.147+0000 7f1ee483f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.147+0000 7f1ee483f640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f1ee00b3810 con 0x7f1ee00b1f30 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.147+0000 7f1ee483f640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.147+0000 7f1ee483f640 10 monclient(hunting): authenticate will time out at 1350.260656s 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.148+0000 7f1ede59c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 0x7f1ee00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.148+0000 7f1ede59c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 0x7f1ee00b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:37198/0 (socket says 172.21.15.19:37198) 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.148+0000 7f1ede59c640 1 -- 172.21.15.19:0/3524670478 learned_addr learned my addr 172.21.15.19:0/3524670478 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.148+0000 7f1ede59c640 10 monclient(hunting): get_auth_request con 0x7f1ee00b1f30 auth_method 0 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.148+0000 7f1ede59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:28.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.148+0000 7f1ede59c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:28.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.148+0000 7f1ede59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:28.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.149+0000 7f1ede59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:28.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.150+0000 7f1ede59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:28.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.150+0000 7f1ede59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:28.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.152+0000 7f1ede59c640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2024-05-04T17:45:28.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.152+0000 7f1ede59c640 1 -- 172.21.15.19:0/3524670478 >> v2:172.21.15.19:3300/0 conn(0x7f1ee01449c0 msgr2=0x7f1ee0151880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:28.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.152+0000 7f1ede59c640 1 --2- 172.21.15.19:0/3524670478 >> v2:172.21.15.19:3300/0 conn(0x7f1ee01449c0 0x7f1ee0151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.153+0000 7f1ede59c640 1 -- 172.21.15.19:0/3524670478 >> v1:172.21.15.19:6789/0 conn(0x7f1ee00b33d0 legacy=0x7f1ee01442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:28.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.153+0000 7f1ede59c640 10 monclient: _finish_hunting 0 2024-05-04T17:45:28.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.153+0000 7f1ede59c640 1 monclient: found mon.noname-b 2024-05-04T17:45:28.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.153+0000 7f1ede59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:28.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.153+0000 7f1ede59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:45:28.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.153+0000 7f1ede59c640 1 -- 172.21.15.19:0/3524670478 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee006f8e0 con 0x7f1ee00b1f30 2024-05-04T17:45:28.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.156+0000 7f1ede59c640 1 --2- 172.21.15.19:0/3524670478 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 0x7f1ee00b2e90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009870 tx=0x7f1ed002b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c75f834d81ea63b server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.162+0000 7f1edd59a640 1 -- 172.21.15.19:0/3524670478 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed002e070 con 0x7f1ee00b1f30 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.162+0000 7f1edd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.162+0000 7f1edd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.162+0000 7f1edd59a640 10 monclient: dump: 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.162+0000 7f1edd59a640 10 monclient: _finish_auth 0 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.162+0000 7f1edd59a640 10 monclient: _check_auth_tickets 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.162+0000 7f1edd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.162+0000 7f1ee483f640 5 monclient: authenticate success, global_id 4304 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.162+0000 7f1ee483f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.163+0000 7f1edd59a640 1 -- 172.21.15.19:0/3524670478 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed0037070 con 0x7f1ee00b1f30 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.163+0000 7f1edd59a640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.163+0000 7f1ee483f640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.163+0000 7f1ee483f640 4 set_mon_vals no callback set 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.163+0000 7f1edd59a640 1 -- 172.21.15.19:0/3524670478 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed002e070 con 0x7f1ee00b1f30 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.163+0000 7f1edd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.163+0000 7f1edd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.163+0000 7f1edd59a640 10 monclient: dump: 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.166 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.167 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.167 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.167 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.167 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.167 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.167 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.167 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.164+0000 7f1ee483f640 10 monclient: shutdown 2024-05-04T17:45:28.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.164+0000 7f1ee483f640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:28.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.164+0000 7f1ee483f640 1 -- 172.21.15.19:0/3524670478 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 msgr2=0x7f1ee00b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.164+0000 7f1ee483f640 1 --2- 172.21.15.19:0/3524670478 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 0x7f1ee00b2e90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009870 tx=0x7f1ed002b880 comp rx=0 tx=0).stop 2024-05-04T17:45:28.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.164+0000 7f1ee483f640 1 -- 172.21.15.19:0/3524670478 shutdown_connections 2024-05-04T17:45:28.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.164+0000 7f1ee483f640 1 --2- 172.21.15.19:0/3524670478 >> v2:172.21.15.19:3300/0 conn(0x7f1ee01449c0 0x7f1ee0151880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.164+0000 7f1ee483f640 1 --2- 172.21.15.19:0/3524670478 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 0x7f1ee00b2e90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.164+0000 7f1ee483f640 1 -- 172.21.15.19:0/3524670478 >> 172.21.15.19:0/3524670478 conn(0x7f1ee0067780 msgr2=0x7f1ee00b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:28.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.165+0000 7f1ee483f640 1 -- 172.21.15.19:0/3524670478 shutdown_connections 2024-05-04T17:45:28.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.165+0000 7f1ee483f640 1 -- 172.21.15.19:0/3524670478 wait complete. 2024-05-04T17:45:28.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.166+0000 7f1ee483f640 10 monclient: build_initial_monmap 2024-05-04T17:45:28.171 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.166+0000 7f1ee483f640 10 monclient: monmap: 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.166+0000 7f1ee483f640 1 librados: starting msgr at 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.166+0000 7f1ee483f640 1 librados: starting objecter 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.167+0000 7f1ee483f640 1 Processor -- start 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.167+0000 7f1ee483f640 1 -- start start 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.167+0000 7f1ee483f640 1 librados: setting wanted keys 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.167+0000 7f1ee483f640 1 librados: calling monclient init 2024-05-04T17:45:28.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.167+0000 7f1ee483f640 10 monclient: init 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.167+0000 7f1ee483f640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.167+0000 7f1ee483f640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 0x7f1ee01d36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 10 monclient(hunting): picked mon.noname-c con 0x7f1ee00b1f30 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1ee00b33d0 0x7f1ee013e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 10 monclient(hunting): picked mon.noname-a con 0x7f1ee00b33d0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ee01449c0 0x7f1ee013ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 10 monclient(hunting): picked mon.noname-b con 0x7f1ee01449c0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee00b4cd0 con 0x7f1ee00b33d0 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee01474d0 con 0x7f1ee01449c0 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f1ee0147d00 con 0x7f1ee00b1f30 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ee483f640 10 monclient(hunting): authenticate will time out at 1350.281797s 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.168+0000 7f1ede59c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 0x7f1ee01d36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.169+0000 7f1ede59c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 0x7f1ee01d36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:37200/0 (socket says 172.21.15.19:37200) 2024-05-04T17:45:28.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.169+0000 7f1ede59c640 1 -- 172.21.15.19:0/152099008 learned_addr learned my addr 172.21.15.19:0/152099008 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:28.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.169+0000 7f1ede59c640 10 monclient(hunting): get_auth_request con 0x7f1ee00b1f30 auth_method 0 2024-05-04T17:45:28.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.169+0000 7f1ede59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:28.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.169+0000 7f1ede59c640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:28.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.169+0000 7f1ede59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:28.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.170+0000 7f1ede59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:28.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.170+0000 7f1ede59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:28.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.170+0000 7f1ede59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:28.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.175+0000 7f1ede59c640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-05-04T17:45:28.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.175+0000 7f1ede59c640 1 -- 172.21.15.19:0/152099008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ee01449c0 msgr2=0x7f1ee013ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.175+0000 7f1ede59c640 1 --2- 172.21.15.19:0/152099008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ee01449c0 0x7f1ee013ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.175+0000 7f1ede59c640 1 -- 172.21.15.19:0/152099008 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1ee00b33d0 msgr2=0x7f1ee013e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.175+0000 7f1ede59c640 1 --2- 172.21.15.19:0/152099008 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1ee00b33d0 0x7f1ee013e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.175+0000 7f1ede59c640 10 monclient: _finish_hunting 0 2024-05-04T17:45:28.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.175+0000 7f1ede59c640 1 monclient: found mon.noname-c 2024-05-04T17:45:28.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.175+0000 7f1ede59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:28.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.175+0000 7f1ede59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T17:45:28.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.175+0000 7f1ede59c640 1 -- 172.21.15.19:0/152099008 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee013f3c0 con 0x7f1ee00b1f30 2024-05-04T17:45:28.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.179+0000 7f1ede59c640 1 --2- 172.21.15.19:0/152099008 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 0x7f1ee01d36c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0000c00 tx=0x7f1ed00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.179+0000 7f1eceffd640 10 client.?.objecter ms_handle_connect 0x7f1ee00b1f30 2024-05-04T17:45:28.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.179+0000 7f1eceffd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:28.183 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-04T17:45:28.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 1 -- 172.21.15.19:0/152099008 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed002e070 con 0x7f1ee00b1f30 2024-05-04T17:45:28.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:28.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 10 monclient: dump: 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 10 monclient: _finish_auth 0 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 10 monclient: _check_auth_tickets 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 1 -- 172.21.15.19:0/152099008 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed0032700 con 0x7f1ee00b1f30 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1ee483f640 5 monclient: authenticate success, global_id 4307 2024-05-04T17:45:28.187 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 1 -- 172.21.15.19:0/152099008 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0032ab0 con 0x7f1ee00b1f30 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1eceffd640 10 monclient: dump: 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1ee483f640 10 monclient: _renew_subs 2024-05-04T17:45:28.188 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1ee483f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:28.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1ee483f640 1 -- 172.21.15.19:0/152099008 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ee01d2c50 con 0x7f1ee00b1f30 2024-05-04T17:45:28.189 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.185+0000 7f1edcd99640 4 set_mon_vals no callback set 2024-05-04T17:45:28.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.186+0000 7f1ee483f640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:28.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.186+0000 7f1ee483f640 10 monclient: _renew_subs 2024-05-04T17:45:28.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.186+0000 7f1ee483f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:28.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.187+0000 7f1ee483f640 1 -- 172.21.15.19:0/152099008 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ee013f610 con 0x7f1ee00b1f30 2024-05-04T17:45:28.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.187+0000 7f1ee483f640 1 librados: init done 2024-05-04T17:45:28.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.188+0000 7f1eccff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.188+0000 7f1eccff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.188+0000 7f1eccff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:28.190 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.188+0000 7f1eccff9640 1 -- 172.21.15.19:0/152099008 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eb8005080 con 0x7f1ee00b1f30 2024-05-04T17:45:28.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.204+0000 7f3d8ddd4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-04T17:45:28.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.204+0000 7f3d8ddd4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-04T17:45:28.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.204+0000 7f3d8ddd4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:28.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.204+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/90103793 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3d880b69a0 con 0x7f3d881404e0 2024-05-04T17:45:28.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.216+0000 7f04f11db640 10 monclient: get_monmap_and_config 2024-05-04T17:45:28.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.216+0000 7f04f11db640 10 monclient: build_initial_monmap 2024-05-04T17:45:28.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.216+0000 7f04f11db640 10 monclient: monmap: 2024-05-04T17:45:28.218 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:28.218 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.218 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:28.217630+0000 2024-05-04T17:45:28.218 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:28.217630+0000 2024-05-04T17:45:28.218 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:28.218 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.218 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:28.218 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:28.218 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:28.218 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:28.218 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:28.218 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.217+0000 7f04f11db640 1 Processor -- start 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.217+0000 7f04f11db640 1 -- start start 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.217+0000 7f04f11db640 10 monclient: init 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f04ec0a9720 0x7f04ec0aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 10 monclient(hunting): picked mon.noname-a con 0x7f04ec0a9720 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04ec0a9b00 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f04ec0b2e90 0x7f04ec13fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 10 monclient(hunting): picked mon.noname-b con 0x7f04ec0b2e90 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ec0abc40 con 0x7f04ec0a9b00 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f04ec146fa0 con 0x7f04ec0a9720 2024-05-04T17:45:28.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f04ec146090 con 0x7f04ec0b2e90 2024-05-04T17:45:28.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:28.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.218+0000 7f04f11db640 10 monclient(hunting): authenticate will time out at 1350.332239s 2024-05-04T17:45:28.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.227+0000 7f1eceffd640 1 -- 172.21.15.19:0/152099008 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1ed0032c50 con 0x7f1ee00b1f30 2024-05-04T17:45:28.228 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.228+0000 7f1eceffd640 1 --2- 172.21.15.19:0/152099008 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f1ec006d7f0 0x7f1ec006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.228+0000 7f1eddd9b640 1 --2- 172.21.15.19:0/152099008 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f1ec006d7f0 0x7f1ec006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.228+0000 7f1eddd9b640 10 monclient: get_auth_request con 0x7f1ec006d7f0 auth_method 0 2024-05-04T17:45:28.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.229+0000 7f1eceffd640 10 client.4307.objecter ms_handle_connect 0x7f1ec006d7f0 2024-05-04T17:45:28.229 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.229+0000 7f1eddd9b640 1 --2- 172.21.15.19:0/152099008 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f1ec006d7f0 0x7f1ec006fcb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1ee01cfa60 tx=0x7f1ed4007990 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.234+0000 7f1eceffd640 1 -- 172.21.15.19:0/152099008 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f1ed00795b0 con 0x7f1ee00b1f30 2024-05-04T17:45:28.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.234+0000 7f1eceffd640 10 client.4307.objecter ms_dispatch 0x7f1ee0000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:28.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.234+0000 7f1eceffd640 3 client.4307.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:28.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.234+0000 7f1eceffd640 3 client.4307.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:28.235 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.234+0000 7f1eceffd640 20 client.4307.objecter dump_active .. 0 homeless 2024-05-04T17:45:28.236 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.235+0000 7f04eb59e640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f04ec0b2e90 0x7f04ec13fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.278 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.278+0000 7fe4137fe640 1 -- 172.21.15.19:0/3879813685 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe414059270 con 0x7fe41c146890 2024-05-04T17:45:28.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.278+0000 7fe4137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.279 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.278+0000 7fe4137fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:28.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.284+0000 7f3d7ffff640 1 -- 172.21.15.19:0/90103793 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3d80010040 con 0x7f3d881404e0 2024-05-04T17:45:28.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.284+0000 7f3d7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-05-04T17:45:28.285 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.284+0000 7f3d7ffff640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:28.285 INFO:teuthology.orchestra.run.smithi019.stdout:81604378633 2024-05-04T17:45:28.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 10 librados: watch_flush enter 2024-05-04T17:45:28.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 10 librados: watch_flush exit 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 20 client.4402.objecter shutdown clearing up homeless session... 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 10 client.4402.objecter successfully canceled tick 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/90103793 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f3d6806d840 msgr2=0x7f3d6806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 1 --2- 172.21.15.19:0/90103793 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f3d6806d840 0x7f3d6806fd00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3d70001780 tx=0x7f3d70009040 comp rx=0 tx=0).stop 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 10 monclient: shutdown 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/90103793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d881404e0 msgr2=0x7f3d8813c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 1 --2- 172.21.15.19:0/90103793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d881404e0 0x7f3d8813c2c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000efc0 tx=0x7f3d8000c490 comp rx=0 tx=0).stop 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/90103793 shutdown_connections 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 1 --2- 172.21.15.19:0/90103793 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f3d6806d840 0x7f3d6806fd00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 1 --2- 172.21.15.19:0/90103793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d881404e0 0x7f3d8813c2c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 1 --2- 172.21.15.19:0/90103793 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f3d880b27f0 0x7f3d8813bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 1 --2- 172.21.15.19:0/90103793 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f3d880a9fb0 0x7f3d88140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.287+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/90103793 >> 172.21.15.19:0/90103793 conn(0x7f3d880b4e90 msgr2=0x7f3d880b5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:28.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.288+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/90103793 shutdown_connections 2024-05-04T17:45:28.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.288+0000 7f3d8ddd4640 1 -- 172.21.15.19:0/90103793 wait complete. 2024-05-04T17:45:28.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.288+0000 7f3d8ddd4640 1 librados: shutdown 2024-05-04T17:45:28.304 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378633 got 81604378633 for osd.4 2024-05-04T17:45:28.304 DEBUG:teuthology.parallel:result is None 2024-05-04T17:45:28.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.306+0000 7fc535738640 10 monclient: get_monmap_and_config 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.306+0000 7fc535738640 10 monclient: build_initial_monmap 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.306+0000 7fc535738640 10 monclient: monmap: 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:28.307091+0000 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:28.307091+0000 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:28.307 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.306+0000 7fc535738640 1 Processor -- start 2024-05-04T17:45:28.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.306+0000 7fc535738640 1 -- start start 2024-05-04T17:45:28.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 10 monclient: init 2024-05-04T17:45:28.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7f04eb59e640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f04ec0b2e90 0x7f04ec13fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:37208/0 (socket says 172.21.15.19:37208) 2024-05-04T17:45:28.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7f04eb59e640 1 -- 172.21.15.19:0/3048115265 learned_addr learned my addr 172.21.15.19:0/3048115265 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:28.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7f04eb59e640 10 monclient(hunting): get_auth_request con 0x7f04ec0b2e90 auth_method 0 2024-05-04T17:45:28.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7f04eb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:28.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7f04eb59e640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:28.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7f04eb59e640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:28.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:28.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5300a9720 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fc5300a9b00 0x7fc5300aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 10 monclient(hunting): picked mon.noname-a con 0x7fc5300a9b00 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc5300b2e90 0x7fc53013fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 10 monclient(hunting): picked mon.noname-c con 0x7fc5300b2e90 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5300abc40 con 0x7fc5300a9720 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fc530146fa0 con 0x7fc5300a9b00 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fc530146090 con 0x7fc5300b2e90 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.307+0000 7fc535738640 10 monclient(hunting): authenticate will time out at 1350.421557s 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.308+0000 7fc52f7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc5300b2e90 0x7fc53013fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.308+0000 7fc52f7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc5300b2e90 0x7fc53013fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36550/0 (socket says 172.21.15.19:36550) 2024-05-04T17:45:28.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.308+0000 7fc52f7fe640 1 -- 172.21.15.19:0/499986620 learned_addr learned my addr 172.21.15.19:0/499986620 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:28.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.308+0000 7fc52f7fe640 10 monclient(hunting): get_auth_request con 0x7fc5300b2e90 auth_method 0 2024-05-04T17:45:28.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.308+0000 7fc52f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:28.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.308+0000 7fc52f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:28.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.308+0000 7fc52f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:28.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.309+0000 7fc52f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:28.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.309+0000 7fc52f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:28.312 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.309+0000 7fc52f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:28.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.310+0000 7fc52f7fe640 10 monclient(hunting): handle_auth_done global_id 4417 payload 274 2024-05-04T17:45:28.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.310+0000 7fc52f7fe640 1 -- 172.21.15.19:0/499986620 >> v2:172.21.15.19:3300/0 conn(0x7fc5300a9b00 msgr2=0x7fc5300aa890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:28.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.310+0000 7fc52f7fe640 1 --2- 172.21.15.19:0/499986620 >> v2:172.21.15.19:3300/0 conn(0x7fc5300a9b00 0x7fc5300aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.310+0000 7fc52f7fe640 1 -- 172.21.15.19:0/499986620 >> v1:172.21.15.19:6789/0 conn(0x7fc5300a9720 legacy=0x7fc530146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:28.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.310+0000 7fc52f7fe640 10 monclient: _finish_hunting 0 2024-05-04T17:45:28.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.310+0000 7fc52f7fe640 1 monclient: found mon.noname-c 2024-05-04T17:45:28.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.310+0000 7fc52f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:28.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.310+0000 7fc52f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:28.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.310+0000 7fc52f7fe640 1 -- 172.21.15.19:0/499986620 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc530146370 con 0x7fc5300b2e90 2024-05-04T17:45:28.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.313+0000 7fc52f7fe640 1 --2- 172.21.15.19:0/499986620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5300b2e90 0x7fc53013fea0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc528009e50 tx=0x7fc52802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bdf17b397a092c server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.316+0000 7fc52dffb640 1 -- 172.21.15.19:0/499986620 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52802ca80 con 0x7fc5300b2e90 2024-05-04T17:45:28.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.316+0000 7fc52dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.316+0000 7fc52dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.316+0000 7fc52dffb640 10 monclient: dump: 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc52dffb640 10 monclient: _finish_auth 0 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc52dffb640 10 monclient: _check_auth_tickets 2024-05-04T17:45:28.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc52dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc52dffb640 1 -- 172.21.15.19:0/499986620 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc52802cc20 con 0x7fc5300b2e90 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc535738640 5 monclient: authenticate success, global_id 4417 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc535738640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc52dffb640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc535738640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc535738640 4 set_mon_vals no callback set 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc52dffb640 1 -- 172.21.15.19:0/499986620 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5280343f0 con 0x7fc5300b2e90 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc52dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc52dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc52dffb640 10 monclient: dump: 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.319 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.320 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.320 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.320 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.320 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.320 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.320 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc535738640 10 monclient: shutdown 2024-05-04T17:45:28.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc535738640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:28.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc535738640 1 -- 172.21.15.19:0/499986620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5300b2e90 msgr2=0x7fc53013fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc535738640 1 --2- 172.21.15.19:0/499986620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5300b2e90 0x7fc53013fea0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc528009e50 tx=0x7fc52802bee0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc535738640 1 -- 172.21.15.19:0/499986620 shutdown_connections 2024-05-04T17:45:28.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc535738640 1 --2- 172.21.15.19:0/499986620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5300b2e90 0x7fc53013fea0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc535738640 1 --2- 172.21.15.19:0/499986620 >> v2:172.21.15.19:3300/0 conn(0x7fc5300a9b00 0x7fc5300aa890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.323 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.317+0000 7fc535738640 1 -- 172.21.15.19:0/499986620 >> 172.21.15.19:0/499986620 conn(0x7fc530067780 msgr2=0x7fc5300b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.318+0000 7fc535738640 1 -- 172.21.15.19:0/499986620 shutdown_connections 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.318+0000 7fc535738640 1 -- 172.21.15.19:0/499986620 wait complete. 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.318+0000 7fc535738640 10 monclient: build_initial_monmap 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.318+0000 7fc535738640 10 monclient: monmap: 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.319+0000 7fc535738640 1 librados: starting msgr at 2024-05-04T17:45:28.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.319+0000 7fc535738640 1 librados: starting objecter 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.319+0000 7fc535738640 1 Processor -- start 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.319+0000 7fc535738640 1 -- start start 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.319+0000 7fc535738640 1 librados: setting wanted keys 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.319+0000 7fc535738640 1 librados: calling monclient init 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.319+0000 7fc535738640 10 monclient: init 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.320+0000 7fc535738640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.320+0000 7fc535738640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.320+0000 7fc535738640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5300a9720 0x7fc530137980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.320+0000 7fc535738640 10 monclient(hunting): picked mon.noname-c con 0x7fc5300a9720 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.320+0000 7fc535738640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5300a9b00 0x7fc5301380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.320+0000 7fc535738640 10 monclient(hunting): picked mon.noname-b con 0x7fc5300a9b00 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.320+0000 7fc535738640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc5300b2e90 0x7fc530131660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.320+0000 7fc535738640 10 monclient(hunting): picked mon.noname-a con 0x7fc5300b2e90 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.320+0000 7fc535738640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.325 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.320+0000 7fc535738640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc530140770 con 0x7fc5300b2e90 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.320+0000 7fc535738640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.321+0000 7fc535738640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc5301405f0 con 0x7fc5300a9b00 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.321+0000 7fc535738640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.321+0000 7fc535738640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc53014d420 con 0x7fc5300a9720 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.321+0000 7fc535738640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.321+0000 7fc535738640 10 monclient(hunting): authenticate will time out at 1350.434736s 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.321+0000 7fc52e7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5300a9b00 0x7fc5301380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.321+0000 7fc52e7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5300a9b00 0x7fc5301380b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36560/0 (socket says 172.21.15.19:36560) 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.321+0000 7fc52e7fc640 1 -- 172.21.15.19:0/735943338 learned_addr learned my addr 172.21.15.19:0/735943338 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.321+0000 7fc52e7fc640 10 monclient(hunting): get_auth_request con 0x7fc5300a9b00 auth_method 0 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.321+0000 7fc52e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.321+0000 7fc52e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.321+0000 7fc52e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.322+0000 7fc52e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:28.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.322+0000 7fc52e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:28.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.322+0000 7fc52e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:28.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7f62d8d84640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-04T17:45:28.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7f62d8d84640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-04T17:45:28.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7f62d8d84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:28.327 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7f62d8d84640 1 -- 172.21.15.19:0/584156358 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f62d4142540 con 0x7f62d4140310 2024-05-04T17:45:28.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7fc52e7fc640 10 monclient(hunting): handle_auth_done global_id 4420 payload 824 2024-05-04T17:45:28.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7fc52e7fc640 1 -- 172.21.15.19:0/735943338 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5300a9720 msgr2=0x7fc530137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7fc52e7fc640 1 --2- 172.21.15.19:0/735943338 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5300a9720 0x7fc530137980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7fc52e7fc640 1 -- 172.21.15.19:0/735943338 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc5300b2e90 msgr2=0x7fc530131660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7fc52e7fc640 1 --2- 172.21.15.19:0/735943338 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc5300b2e90 0x7fc530131660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.328 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7fc52e7fc640 10 monclient: _finish_hunting 0 2024-05-04T17:45:28.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7fc52e7fc640 1 monclient: found mon.noname-b 2024-05-04T17:45:28.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7fc52e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:28.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7fc52e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:28.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.324+0000 7fc52e7fc640 1 -- 172.21.15.19:0/735943338 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc530140ca0 con 0x7fc5300a9b00 2024-05-04T17:45:28.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.327+0000 7fc52e7fc640 1 --2- 172.21.15.19:0/735943338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5300a9b00 0x7fc5301380b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc52400b4a0 tx=0x7fc52400b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.327+0000 7fc5177fe640 10 client.?.objecter ms_handle_connect 0x7fc5300a9b00 2024-05-04T17:45:28.329 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.327+0000 7fc5177fe640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:28.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 1 -- 172.21.15.19:0/735943338 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc524014070 con 0x7fc5300a9b00 2024-05-04T17:45:28.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:28.332 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 10 monclient: dump: 2024-05-04T17:45:28.332 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.332 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.332 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.332 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.332 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.332 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.332 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.332 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 1 -- 172.21.15.19:0/735943338 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc524010610 con 0x7fc5300a9b00 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 1 -- 172.21.15.19:0/735943338 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5240109e0 con 0x7fc5300a9b00 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc5177fe640 10 monclient: dump: 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.333 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc535738640 5 monclient: authenticate success, global_id 4420 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc52d7fa640 4 set_mon_vals no callback set 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc535738640 10 monclient: _renew_subs 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc535738640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc535738640 1 -- 172.21.15.19:0/735943338 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5301387d0 con 0x7fc5300a9b00 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc535738640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc535738640 10 monclient: _renew_subs 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc535738640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:28.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc535738640 1 -- 172.21.15.19:0/735943338 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc530131ba0 con 0x7fc5300a9b00 2024-05-04T17:45:28.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.330+0000 7fc535738640 1 librados: init done 2024-05-04T17:45:28.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.331+0000 7fc535738640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.331+0000 7fc535738640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.331+0000 7fc535738640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:28.335 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.331+0000 7fc535738640 1 -- 172.21.15.19:0/735943338 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5300b33b0 con 0x7fc5300a9b00 2024-05-04T17:45:28.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.350+0000 7fc5177fe640 1 -- 172.21.15.19:0/735943338 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc524017070 con 0x7fc5300a9b00 2024-05-04T17:45:28.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.351+0000 7fc5177fe640 1 --2- 172.21.15.19:0/735943338 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc51c06d7c0 0x7fc51c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.351+0000 7fc52effd640 1 --2- 172.21.15.19:0/735943338 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc51c06d7c0 0x7fc51c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.351+0000 7fc52effd640 10 monclient: get_auth_request con 0x7fc51c06d7c0 auth_method 0 2024-05-04T17:45:28.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.351+0000 7fc52effd640 1 --2- 172.21.15.19:0/735943338 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc51c06d7c0 0x7fc51c06fc80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc520001560 tx=0x7fc520007990 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.351+0000 7fc5177fe640 10 client.4420.objecter ms_handle_connect 0x7fc51c06d7c0 2024-05-04T17:45:28.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.373+0000 7f5f0f7fe640 1 -- 172.21.15.19:0/529660183 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5f08021020 con 0x7f5f180a9720 2024-05-04T17:45:28.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.373+0000 7f5f0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.374 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.373+0000 7f5f0f7fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:28.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.388+0000 7f04ead9d640 1 --2- 172.21.15.19:0/3048115265 >> v2:172.21.15.19:3300/0 conn(0x7f04ec0a9720 0x7f04ec0aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.390+0000 7f04ead9d640 10 monclient(hunting): get_auth_request con 0x7f04ec0a9720 auth_method 0 2024-05-04T17:45:28.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.390+0000 7f04ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:28.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.390+0000 7f04ead9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:28.391 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.390+0000 7f04ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:28.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.394+0000 7f04ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:28.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.394+0000 7f04ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:28.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.394+0000 7f04ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:28.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.397+0000 7f04ead9d640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2024-05-04T17:45:28.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.397+0000 7f04ead9d640 1 -- 172.21.15.19:0/3048115265 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f04ec0b2e90 msgr2=0x7f04ec13fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:28.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.397+0000 7f04ead9d640 1 --2- 172.21.15.19:0/3048115265 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f04ec0b2e90 0x7f04ec13fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.398 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.397+0000 7f04ead9d640 1 -- 172.21.15.19:0/3048115265 >> v1:172.21.15.19:6789/0 conn(0x7f04ec0a9b00 legacy=0x7f04ec146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:28.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.397+0000 7f04ead9d640 10 monclient: _finish_hunting 0 2024-05-04T17:45:28.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.397+0000 7f04ead9d640 1 monclient: found mon.noname-a 2024-05-04T17:45:28.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.397+0000 7f04ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:28.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.397+0000 7f04ead9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:28.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.397+0000 7f04ead9d640 1 -- 172.21.15.19:0/3048115265 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec146370 con 0x7f04ec0a9720 2024-05-04T17:45:28.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.404+0000 7f04ead9d640 1 --2- 172.21.15.19:0/3048115265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04ec0a9720 0x7f04ec0aa890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0097a0 tx=0x7f04dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caee13dc7cfef3e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 1 -- 172.21.15.19:0/3048115265 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc0315a0 con 0x7f04ec0a9720 2024-05-04T17:45:28.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:28.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 10 monclient: dump: 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 10 monclient: _finish_auth 0 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 10 monclient: _check_auth_tickets 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 1 -- 172.21.15.19:0/3048115265 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04dc037070 con 0x7f04ec0a9720 2024-05-04T17:45:28.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04f11db640 5 monclient: authenticate success, global_id 4335 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04f11db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04f11db640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 1 -- 172.21.15.19:0/3048115265 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc031960 con 0x7f04ec0a9720 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04f11db640 4 set_mon_vals no callback set 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.413+0000 7f04e9d9b640 10 monclient: dump: 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.417 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.414+0000 7f04f11db640 10 monclient: shutdown 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.414+0000 7f04f11db640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.414+0000 7f04f11db640 1 -- 172.21.15.19:0/3048115265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04ec0a9720 msgr2=0x7f04ec0aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.414+0000 7f04f11db640 1 --2- 172.21.15.19:0/3048115265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04ec0a9720 0x7f04ec0aa890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0097a0 tx=0x7f04dc02b840 comp rx=0 tx=0).stop 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.414+0000 7f04f11db640 1 -- 172.21.15.19:0/3048115265 shutdown_connections 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.414+0000 7f04f11db640 1 --2- 172.21.15.19:0/3048115265 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f04ec0b2e90 0x7f04ec13fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.414+0000 7f04f11db640 1 --2- 172.21.15.19:0/3048115265 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04ec0a9720 0x7f04ec0aa890 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.414+0000 7f04f11db640 1 -- 172.21.15.19:0/3048115265 >> 172.21.15.19:0/3048115265 conn(0x7f04ec067780 msgr2=0x7f04ec0b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:28.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.414+0000 7f04f11db640 1 -- 172.21.15.19:0/3048115265 shutdown_connections 2024-05-04T17:45:28.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.414+0000 7f04f11db640 1 -- 172.21.15.19:0/3048115265 wait complete. 2024-05-04T17:45:28.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.415+0000 7f04f11db640 10 monclient: build_initial_monmap 2024-05-04T17:45:28.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.415+0000 7f04f11db640 10 monclient: monmap: 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.415+0000 7f04f11db640 1 librados: starting msgr at 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.415+0000 7f04f11db640 1 librados: starting objecter 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.415+0000 7f04f11db640 1 Processor -- start 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.415+0000 7f04f11db640 1 -- start start 2024-05-04T17:45:28.421 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.415+0000 7f04f11db640 1 librados: setting wanted keys 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.415+0000 7f04f11db640 1 librados: calling monclient init 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.415+0000 7f04f11db640 10 monclient: init 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.416+0000 7f04f11db640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.416+0000 7f04f11db640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.416+0000 7f04f11db640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04ec0a9720 0x7f04ec0acc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.416+0000 7f04f11db640 10 monclient(hunting): picked mon.noname-a con 0x7f04ec0a9720 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.416+0000 7f04f11db640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f04ec0a9b00 0x7f04ec1396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04f11db640 10 monclient(hunting): picked mon.noname-c con 0x7f04ec0a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04f11db640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04ec0b2e90 0x7f04ec139c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04f11db640 10 monclient(hunting): picked mon.noname-b con 0x7f04ec0b2e90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04f11db640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04f11db640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec1407b0 con 0x7f04ec0a9720 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04f11db640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.422 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04f11db640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec140630 con 0x7f04ec0b2e90 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04f11db640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04f11db640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f04ec14d420 con 0x7f04ec0a9b00 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04f11db640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04f11db640 10 monclient(hunting): authenticate will time out at 1350.530693s 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04ead9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04ec0a9720 0x7f04ec0acc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04ead9d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04ec0a9720 0x7f04ec0acc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:37482/0 (socket says 172.21.15.19:37482) 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04ead9d640 1 -- 172.21.15.19:0/686012755 learned_addr learned my addr 172.21.15.19:0/686012755 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04ead9d640 10 monclient(hunting): get_auth_request con 0x7f04ec0a9720 auth_method 0 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04ead9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.417+0000 7f04ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.418+0000 7f04ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.418+0000 7f04ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:28.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.418+0000 7f04ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:28.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.421+0000 7f04ead9d640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2024-05-04T17:45:28.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.421+0000 7f04ead9d640 1 -- 172.21.15.19:0/686012755 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f04ec0a9b00 msgr2=0x7f04ec1396e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.421+0000 7f04ead9d640 1 --2- 172.21.15.19:0/686012755 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f04ec0a9b00 0x7f04ec1396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.421+0000 7f04ead9d640 1 -- 172.21.15.19:0/686012755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04ec0b2e90 msgr2=0x7f04ec139c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.421+0000 7f04ead9d640 1 --2- 172.21.15.19:0/686012755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04ec0b2e90 0x7f04ec139c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.421+0000 7f04ead9d640 10 monclient: _finish_hunting 0 2024-05-04T17:45:28.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.421+0000 7f04ead9d640 1 monclient: found mon.noname-a 2024-05-04T17:45:28.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.421+0000 7f04ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:28.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.421+0000 7f04ead9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:28.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.421+0000 7f04ead9d640 1 -- 172.21.15.19:0/686012755 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec0aed60 con 0x7f04ec0a9720 2024-05-04T17:45:28.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.424+0000 7f04ead9d640 1 --2- 172.21.15.19:0/686012755 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04ec0a9720 0x7f04ec0acc20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f04dc000c00 tx=0x7f04dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.424+0000 7f04d37fe640 10 client.?.objecter ms_handle_connect 0x7f04ec0a9720 2024-05-04T17:45:28.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.424+0000 7f04d37fe640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:28.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 1 -- 172.21.15.19:0/686012755 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc032840 con 0x7f04ec0a9720 2024-05-04T17:45:28.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 10 monclient: dump: 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:28.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 1 -- 172.21.15.19:0/686012755 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04dc0329e0 con 0x7f04ec0a9720 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04f11db640 5 monclient: authenticate success, global_id 4338 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04f11db640 10 monclient: _renew_subs 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04f11db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04f11db640 1 -- 172.21.15.19:0/686012755 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ec0af010 con 0x7f04ec0a9720 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 1 -- 172.21.15.19:0/686012755 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc032dd0 con 0x7f04ec0a9720 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04d37fe640 10 monclient: dump: 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.432 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:28.433 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:28.433 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:28.433 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:28.433 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:28.433 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:28.433 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:28.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04f11db640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:28.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04e959a640 4 set_mon_vals no callback set 2024-05-04T17:45:28.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04f11db640 10 monclient: _renew_subs 2024-05-04T17:45:28.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04f11db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:28.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04f11db640 1 -- 172.21.15.19:0/686012755 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ec0ad160 con 0x7f04ec0a9720 2024-05-04T17:45:28.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.429+0000 7f04f11db640 1 librados: init done 2024-05-04T17:45:28.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.430+0000 7f04f11db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.430+0000 7f04f11db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.430+0000 7f04f11db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:28.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.430+0000 7f04f11db640 1 -- 172.21.15.19:0/686012755 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ec06a700 con 0x7f04ec0a9720 2024-05-04T17:45:28.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.467+0000 7f04d37fe640 1 -- 172.21.15.19:0/686012755 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f04dc03b3f0 con 0x7f04ec0a9720 2024-05-04T17:45:28.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.468+0000 7f04d37fe640 1 --2- 172.21.15.19:0/686012755 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f04d806d810 0x7f04d806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:28.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.468+0000 7f04ea59c640 1 --2- 172.21.15.19:0/686012755 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f04d806d810 0x7f04d806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:28.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.468+0000 7f04ea59c640 10 monclient: get_auth_request con 0x7f04d806d810 auth_method 0 2024-05-04T17:45:28.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.469+0000 7f04d37fe640 10 client.4338.objecter ms_handle_connect 0x7f04d806d810 2024-05-04T17:45:28.469 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.469+0000 7f04ea59c640 1 --2- 172.21.15.19:0/686012755 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f04d806d810 0x7f04d806fcd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0ac570 tx=0x7f04e0007990 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:28.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.529+0000 7fe4215bd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-04T17:45:28.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.529+0000 7fe4215bd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-04T17:45:28.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.529+0000 7fe4215bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:28.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.529+0000 7fe4215bd640 1 -- 172.21.15.19:0/3879813685 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fe41c149d70 con 0x7fe41c146890 2024-05-04T17:45:28.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.536+0000 7f1eceffd640 1 -- 172.21.15.19:0/152099008 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1ed0040070 con 0x7f1ee00b1f30 2024-05-04T17:45:28.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.536+0000 7f1eceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.537 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.536+0000 7f1eceffd640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:28.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.550+0000 7f62c2ffd640 1 -- 172.21.15.19:0/584156358 <== mon.2 v2:172.21.15.19:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f62c405d5f0 con 0x7f62d4140310 2024-05-04T17:45:28.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.550+0000 7f62c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-05-04T17:45:28.551 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.550+0000 7f62c2ffd640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:28.551 INFO:teuthology.orchestra.run.smithi019.stdout:68719476744 2024-05-04T17:45:28.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.552+0000 7f62d8d84640 10 librados: watch_flush enter 2024-05-04T17:45:28.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.552+0000 7f62d8d84640 10 librados: watch_flush exit 2024-05-04T17:45:28.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 20 client.4301.objecter shutdown clearing up homeless session... 2024-05-04T17:45:28.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 10 client.4301.objecter successfully canceled tick 2024-05-04T17:45:28.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 -- 172.21.15.19:0/584156358 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f62b406d810 msgr2=0x7f62b406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 --2- 172.21.15.19:0/584156358 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f62b406d810 0x7f62b406fcd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f62cc036000 tx=0x7f62cc030000 comp rx=0 tx=0).stop 2024-05-04T17:45:28.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 10 monclient: shutdown 2024-05-04T17:45:28.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:28.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 -- 172.21.15.19:0/584156358 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d4140310 msgr2=0x7f62d41416b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 --2- 172.21.15.19:0/584156358 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d4140310 0x7f62d41416b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f62c400cbc0 tx=0x7f62c40021c0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 -- 172.21.15.19:0/584156358 shutdown_connections 2024-05-04T17:45:28.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 --2- 172.21.15.19:0/584156358 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f62b406d810 0x7f62b406fcd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 --2- 172.21.15.19:0/584156358 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f62d4140310 0x7f62d41416b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 --2- 172.21.15.19:0/584156358 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f62d40b27f0 0x7f62d41444e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 --2- 172.21.15.19:0/584156358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62d40a9fb0 0x7f62d4143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 -- 172.21.15.19:0/584156358 >> 172.21.15.19:0/584156358 conn(0x7f62d40b4e90 msgr2=0x7f62d40b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:28.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 -- 172.21.15.19:0/584156358 shutdown_connections 2024-05-04T17:45:28.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 -- 172.21.15.19:0/584156358 wait complete. 2024-05-04T17:45:28.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.553+0000 7f62d8d84640 1 librados: shutdown 2024-05-04T17:45:28.565 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476744 got 68719476744 for osd.3 2024-05-04T17:45:28.565 DEBUG:teuthology.parallel:result is None 2024-05-04T17:45:28.586 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.586+0000 7fc5177fe640 1 -- 172.21.15.19:0/735943338 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7fc524057050 con 0x7fc5300a9b00 2024-05-04T17:45:28.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.586+0000 7fc5177fe640 10 client.4420.objecter ms_dispatch 0x7fc530000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:28.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.586+0000 7fc5177fe640 3 client.4420.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:28.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.586+0000 7fc5177fe640 3 client.4420.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:28.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.586+0000 7fc5177fe640 20 client.4420.objecter dump_active .. 0 homeless 2024-05-04T17:45:28.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.600+0000 7fc5177fe640 1 -- 172.21.15.19:0/735943338 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc5300b33b0 con 0x7fc5300a9b00 2024-05-04T17:45:28.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.600+0000 7fc5177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.600+0000 7fc5177fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:28.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.622+0000 7fe4137fe640 1 -- 172.21.15.19:0/3879813685 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe41405d570 con 0x7fe41c146890 2024-05-04T17:45:28.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.622+0000 7fe4137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-05-04T17:45:28.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.622+0000 7fe4137fe640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:28.623 INFO:teuthology.orchestra.run.smithi019.stdout:55834574854 2024-05-04T17:45:28.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 10 librados: watch_flush enter 2024-05-04T17:45:28.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 10 librados: watch_flush exit 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 20 client.4414.objecter shutdown clearing up homeless session... 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 10 client.4414.objecter successfully canceled tick 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 -- 172.21.15.19:0/3879813685 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe3fc06d7c0 msgr2=0x7fe3fc06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe3fc06d7c0 0x7fe3fc06fc80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe40c009560 tx=0x7fe40c008030 comp rx=0 tx=0).stop 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 10 monclient: shutdown 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 -- 172.21.15.19:0/3879813685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe41c146890 msgr2=0x7fe41c139740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe41c146890 0x7fe41c139740 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe41400caa0 tx=0x7fe41400cea0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 -- 172.21.15.19:0/3879813685 shutdown_connections 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe3fc06d7c0 0x7fe3fc06fc80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe41c146890 0x7fe41c139740 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe41c0a9b00 0x7fe41c139200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 --2- 172.21.15.19:0/3879813685 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe41c0a9720 0x7fe41c14a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 -- 172.21.15.19:0/3879813685 >> 172.21.15.19:0/3879813685 conn(0x7fe41c067780 msgr2=0x7fe41c0aa210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:28.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 -- 172.21.15.19:0/3879813685 shutdown_connections 2024-05-04T17:45:28.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 -- 172.21.15.19:0/3879813685 wait complete. 2024-05-04T17:45:28.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.625+0000 7fe4215bd640 1 librados: shutdown 2024-05-04T17:45:28.643 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-05-04T17:45:28.643 DEBUG:teuthology.parallel:result is None 2024-05-04T17:45:28.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.658+0000 7f5f1d54b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-04T17:45:28.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.658+0000 7f5f1d54b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-04T17:45:28.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.658+0000 7f5f1d54b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:28.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.658+0000 7f5f1d54b640 1 -- 172.21.15.19:0/529660183 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f5f1813b2e0 con 0x7f5f180a9720 2024-05-04T17:45:28.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.741+0000 7f1eccff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-04T17:45:28.742 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.741+0000 7f1eccff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-04T17:45:28.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.741+0000 7f1eccff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T17:45:28.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.741+0000 7f1eccff9640 1 -- 172.21.15.19:0/152099008 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f1eb8005640 con 0x7f1ee00b1f30 2024-05-04T17:45:28.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.747+0000 7f1eceffd640 1 -- 172.21.15.19:0/152099008 <== mon.2 v2:172.21.15.19:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1ed007bc10 con 0x7f1ee00b1f30 2024-05-04T17:45:28.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.747+0000 7f1eceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-05-04T17:45:28.748 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.747+0000 7f1eceffd640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:28.748 INFO:teuthology.orchestra.run.smithi019.stdout:81604378632 2024-05-04T17:45:28.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 10 librados: watch_flush enter 2024-05-04T17:45:28.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 10 librados: watch_flush exit 2024-05-04T17:45:28.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 20 client.4307.objecter shutdown clearing up homeless session... 2024-05-04T17:45:28.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 10 client.4307.objecter successfully canceled tick 2024-05-04T17:45:28.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 1 -- 172.21.15.19:0/152099008 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f1ec006d7f0 msgr2=0x7f1ec006fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 1 --2- 172.21.15.19:0/152099008 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f1ec006d7f0 0x7f1ec006fcb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1ee01cfa60 tx=0x7f1ed4007990 comp rx=0 tx=0).stop 2024-05-04T17:45:28.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 10 monclient: shutdown 2024-05-04T17:45:28.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:28.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 1 -- 172.21.15.19:0/152099008 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 msgr2=0x7f1ee01d36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 1 --2- 172.21.15.19:0/152099008 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 0x7f1ee01d36c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0000c00 tx=0x7f1ed00046c0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 1 -- 172.21.15.19:0/152099008 shutdown_connections 2024-05-04T17:45:28.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 1 --2- 172.21.15.19:0/152099008 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f1ec006d7f0 0x7f1ec006fcb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 1 --2- 172.21.15.19:0/152099008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ee01449c0 0x7f1ee013ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 1 --2- 172.21.15.19:0/152099008 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1ee00b33d0 0x7f1ee013e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 1 --2- 172.21.15.19:0/152099008 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1ee00b1f30 0x7f1ee01d36c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 1 -- 172.21.15.19:0/152099008 >> 172.21.15.19:0/152099008 conn(0x7f1ee0067780 msgr2=0x7f1ee00b83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:28.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 1 -- 172.21.15.19:0/152099008 shutdown_connections 2024-05-04T17:45:28.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.749+0000 7f1eccff9640 1 -- 172.21.15.19:0/152099008 wait complete. 2024-05-04T17:45:28.752 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.750+0000 7f1eccff9640 1 librados: shutdown 2024-05-04T17:45:28.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378632 got 81604378632 for osd.6 2024-05-04T17:45:28.762 DEBUG:teuthology.parallel:result is None 2024-05-04T17:45:28.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.817+0000 7f04d37fe640 1 -- 172.21.15.19:0/686012755 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f04dc02f070 con 0x7f04ec0a9720 2024-05-04T17:45:28.818 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.817+0000 7f04d37fe640 10 client.4338.objecter ms_dispatch 0x7f04ec000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:28.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.817+0000 7f04d37fe640 3 client.4338.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:28.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.817+0000 7f04d37fe640 3 client.4338.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:28.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.817+0000 7f04d37fe640 20 client.4338.objecter dump_active .. 0 homeless 2024-05-04T17:45:28.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.817+0000 7f04d37fe640 1 -- 172.21.15.19:0/686012755 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f04dc040070 con 0x7f04ec0a9720 2024-05-04T17:45:28.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.817+0000 7f04d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:28.819 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.817+0000 7f04d37fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:28.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.821+0000 7fc535738640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-04T17:45:28.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.821+0000 7fc535738640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-04T17:45:28.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.821+0000 7fc535738640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:28.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.821+0000 7fc535738640 1 -- 172.21.15.19:0/735943338 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc5301385f0 con 0x7fc5300a9b00 2024-05-04T17:45:28.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.825+0000 7fc5177fe640 1 -- 172.21.15.19:0/735943338 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc5301385f0 con 0x7fc5300a9b00 2024-05-04T17:45:28.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.825+0000 7fc5177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-05-04T17:45:28.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.825+0000 7fc5177fe640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:28.826 INFO:teuthology.orchestra.run.smithi019.stdout:90194313226 2024-05-04T17:45:28.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 10 librados: watch_flush enter 2024-05-04T17:45:28.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 10 librados: watch_flush exit 2024-05-04T17:45:28.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 20 client.4420.objecter shutdown clearing up homeless session... 2024-05-04T17:45:28.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 10 client.4420.objecter successfully canceled tick 2024-05-04T17:45:28.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 -- 172.21.15.19:0/735943338 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc51c06d7c0 msgr2=0x7fc51c06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 --2- 172.21.15.19:0/735943338 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc51c06d7c0 0x7fc51c06fc80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc520001560 tx=0x7fc520007990 comp rx=0 tx=0).stop 2024-05-04T17:45:28.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 10 monclient: shutdown 2024-05-04T17:45:28.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:28.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 -- 172.21.15.19:0/735943338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5300a9b00 msgr2=0x7fc5301380b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 --2- 172.21.15.19:0/735943338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5300a9b00 0x7fc5301380b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc52400b4a0 tx=0x7fc52400b960 comp rx=0 tx=0).stop 2024-05-04T17:45:28.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 -- 172.21.15.19:0/735943338 shutdown_connections 2024-05-04T17:45:28.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 --2- 172.21.15.19:0/735943338 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc51c06d7c0 0x7fc51c06fc80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 --2- 172.21.15.19:0/735943338 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc5300b2e90 0x7fc530131660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 --2- 172.21.15.19:0/735943338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5300a9b00 0x7fc5301380b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 --2- 172.21.15.19:0/735943338 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5300a9720 0x7fc530137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 -- 172.21.15.19:0/735943338 >> 172.21.15.19:0/735943338 conn(0x7fc530067780 msgr2=0x7fc5300aa210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:28.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 -- 172.21.15.19:0/735943338 shutdown_connections 2024-05-04T17:45:28.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 -- 172.21.15.19:0/735943338 wait complete. 2024-05-04T17:45:28.831 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.827+0000 7fc535738640 1 librados: shutdown 2024-05-04T17:45:28.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313225 got 90194313226 for osd.1 2024-05-04T17:45:28.840 DEBUG:teuthology.parallel:result is None 2024-05-04T17:45:28.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.981+0000 7f5f0f7fe640 1 -- 172.21.15.19:0/529660183 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5f080588f0 con 0x7f5f180a9720 2024-05-04T17:45:28.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.981+0000 7f5f0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-05-04T17:45:28.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.981+0000 7f5f0f7fe640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:28.982 INFO:teuthology.orchestra.run.smithi019.stdout:85899345929 2024-05-04T17:45:28.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 10 librados: watch_flush enter 2024-05-04T17:45:28.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 10 librados: watch_flush exit 2024-05-04T17:45:28.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 20 client.4329.objecter shutdown clearing up homeless session... 2024-05-04T17:45:28.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 10 client.4329.objecter successfully canceled tick 2024-05-04T17:45:28.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 -- 172.21.15.19:0/529660183 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5ef806d840 msgr2=0x7f5ef806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5ef806d840 0x7f5ef806fd00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5f00036000 tx=0x7f5f00030000 comp rx=0 tx=0).stop 2024-05-04T17:45:28.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 10 monclient: shutdown 2024-05-04T17:45:28.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:28.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 -- 172.21.15.19:0/529660183 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5f180a9720 msgr2=0x7f5f18147a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:28.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5f180a9720 0x7f5f18147a70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5f0800c3f0 tx=0x7f5f0800c8b0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 -- 172.21.15.19:0/529660183 shutdown_connections 2024-05-04T17:45:28.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5ef806d840 0x7f5ef806fd00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f180b2e90 0x7f5f1814cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5f180a9b00 0x7f5f1814c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 --2- 172.21.15.19:0/529660183 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5f180a9720 0x7f5f18147a70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:28.988 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 -- 172.21.15.19:0/529660183 >> 172.21.15.19:0/529660183 conn(0x7f5f18067780 msgr2=0x7f5f180aa210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:28.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 -- 172.21.15.19:0/529660183 shutdown_connections 2024-05-04T17:45:28.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 -- 172.21.15.19:0/529660183 wait complete. 2024-05-04T17:45:28.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:28.983+0000 7f5f1d54b640 1 librados: shutdown 2024-05-04T17:45:28.996 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345929 for osd.0 2024-05-04T17:45:28.996 DEBUG:teuthology.parallel:result is None 2024-05-04T17:45:29.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.034+0000 7f04f11db640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-04T17:45:29.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.034+0000 7f04f11db640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-04T17:45:29.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.034+0000 7f04f11db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:45:29.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.034+0000 7f04f11db640 1 -- 172.21.15.19:0/686012755 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f04ec13b9c0 con 0x7f04ec0a9720 2024-05-04T17:45:29.038 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.037+0000 7f04d37fe640 1 -- 172.21.15.19:0/686012755 <== mon.0 v2:172.21.15.19:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f04dc07ba90 con 0x7f04ec0a9720 2024-05-04T17:45:29.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.037+0000 7f04d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-05-04T17:45:29.039 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.038+0000 7f04d37fe640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:29.039 INFO:teuthology.orchestra.run.smithi019.stdout:60129542151 2024-05-04T17:45:29.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.039+0000 7f04f11db640 10 librados: watch_flush enter 2024-05-04T17:45:29.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.039+0000 7f04f11db640 10 librados: watch_flush exit 2024-05-04T17:45:29.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.039+0000 7f04f11db640 20 client.4338.objecter shutdown clearing up homeless session... 2024-05-04T17:45:29.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.039+0000 7f04f11db640 10 client.4338.objecter successfully canceled tick 2024-05-04T17:45:29.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.039+0000 7f04f11db640 1 -- 172.21.15.19:0/686012755 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f04d806d810 msgr2=0x7f04d806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.043 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.039+0000 7f04f11db640 1 --2- 172.21.15.19:0/686012755 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f04d806d810 0x7f04d806fcd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0ac570 tx=0x7f04e0007990 comp rx=0 tx=0).stop 2024-05-04T17:45:29.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.039+0000 7f04f11db640 10 monclient: shutdown 2024-05-04T17:45:29.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.039+0000 7f04f11db640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:29.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.039+0000 7f04f11db640 1 -- 172.21.15.19:0/686012755 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04ec0a9720 msgr2=0x7f04ec0acc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.039+0000 7f04f11db640 1 --2- 172.21.15.19:0/686012755 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04ec0a9720 0x7f04ec0acc20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f04dc000c00 tx=0x7f04dc0046c0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.040+0000 7f04f11db640 1 -- 172.21.15.19:0/686012755 shutdown_connections 2024-05-04T17:45:29.044 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.040+0000 7f04f11db640 1 --2- 172.21.15.19:0/686012755 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f04d806d810 0x7f04d806fcd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.040+0000 7f04f11db640 1 --2- 172.21.15.19:0/686012755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04ec0b2e90 0x7f04ec139c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.040+0000 7f04f11db640 1 --2- 172.21.15.19:0/686012755 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f04ec0a9b00 0x7f04ec1396e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.040+0000 7f04f11db640 1 --2- 172.21.15.19:0/686012755 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f04ec0a9720 0x7f04ec0acc20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.040+0000 7f04f11db640 1 -- 172.21.15.19:0/686012755 >> 172.21.15.19:0/686012755 conn(0x7f04ec067780 msgr2=0x7f04ec13b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:29.045 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.040+0000 7f04f11db640 1 -- 172.21.15.19:0/686012755 shutdown_connections 2024-05-04T17:45:29.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.040+0000 7f04f11db640 1 -- 172.21.15.19:0/686012755 wait complete. 2024-05-04T17:45:29.046 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.040+0000 7f04f11db640 1 librados: shutdown 2024-05-04T17:45:29.052 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.2 2024-05-04T17:45:29.052 DEBUG:teuthology.parallel:result is None 2024-05-04T17:45:29.052 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-04T17:45:29.053 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T17:45:29.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.191+0000 7f949c6a5640 10 monclient: get_monmap_and_config 2024-05-04T17:45:29.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.191+0000 7f949c6a5640 10 monclient: build_initial_monmap 2024-05-04T17:45:29.192 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.191+0000 7f949c6a5640 10 monclient: monmap: 2024-05-04T17:45:29.192 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:29.192 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:29.192 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:29.192599+0000 2024-05-04T17:45:29.192 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:29.192599+0000 2024-05-04T17:45:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:29.193 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:29.193 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:29.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.192+0000 7f949c6a5640 1 Processor -- start 2024-05-04T17:45:29.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.192+0000 7f949c6a5640 1 -- start start 2024-05-04T17:45:29.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.192+0000 7f949c6a5640 10 monclient: init 2024-05-04T17:45:29.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:29.195 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9494071b20 0x7f9494071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 10 monclient(hunting): picked mon.noname-b con 0x7f9494071b20 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9494072440 0x7f949413fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 10 monclient(hunting): picked mon.noname-c con 0x7f9494072440 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9494140200 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9494071120 con 0x7f9494140200 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f9494071390 con 0x7f9494072440 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f949406c4c0 con 0x7f9494071b20 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949c6a5640 10 monclient(hunting): authenticate will time out at 1351.306825s 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f9499c40640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9494072440 0x7f949413fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949a441640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9494071b20 0x7f9494071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949ac42640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f9494140200 0x7f94941430d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:42422/0 (socket says 172.21.15.19:42422) 2024-05-04T17:45:29.196 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.193+0000 7f949ac42640 1 -- 172.21.15.19:0/809874935 learned_addr learned my addr 172.21.15.19:0/809874935 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f9499c40640 10 monclient(hunting): get_auth_request con 0x7f9494072440 auth_method 0 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f9499c40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f9499c40640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f9499c40640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f949a441640 10 monclient(hunting): get_auth_request con 0x7f9494071b20 auth_method 0 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f949a441640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f949a441640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f949a441640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f9499c40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f9499c40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f9499c40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f949a441640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f949a441640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:29.197 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.194+0000 7f949a441640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:29.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.196+0000 7f9499c40640 10 monclient(hunting): handle_auth_done global_id 4423 payload 274 2024-05-04T17:45:29.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.196+0000 7f9499c40640 1 -- 172.21.15.19:0/809874935 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9494071b20 msgr2=0x7f9494071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:29.198 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.196+0000 7f9499c40640 1 --2- 172.21.15.19:0/809874935 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9494071b20 0x7f9494071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.196+0000 7f9499c40640 1 -- 172.21.15.19:0/809874935 >> v1:172.21.15.19:6789/0 conn(0x7f9494140200 legacy=0x7f94941430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.196+0000 7f9499c40640 10 monclient: _finish_hunting 0 2024-05-04T17:45:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.196+0000 7f9499c40640 1 monclient: found mon.noname-c 2024-05-04T17:45:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.196+0000 7f9499c40640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.196+0000 7f9499c40640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T17:45:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.196+0000 7f9499c40640 1 -- 172.21.15.19:0/809874935 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94941437e0 con 0x7f9494072440 2024-05-04T17:45:29.199 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.198+0000 7f9499c40640 1 --2- 172.21.15.19:0/809874935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9494072440 0x7f949413fcc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f94880097a0 tx=0x7f948802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efdd7782aa301b66 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:29.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 1 -- 172.21.15.19:0/809874935 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9488035020 con 0x7f9494072440 2024-05-04T17:45:29.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:29.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T17:45:29.203 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 10 monclient: dump: 2024-05-04T17:45:29.203 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:29.203 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:29.203 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.203 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.203 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:29.203 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:29.203 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:29.203 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 10 monclient: _finish_auth 0 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 10 monclient: _check_auth_tickets 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 1 -- 172.21.15.19:0/809874935 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9488031830 con 0x7f9494072440 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949c6a5640 5 monclient: authenticate success, global_id 4423 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949c6a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949c6a5640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 1 -- 172.21.15.19:0/809874935 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9488031c00 con 0x7f9494072440 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949c6a5640 4 set_mon_vals no callback set 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:29.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949943f640 10 monclient: dump: 2024-05-04T17:45:29.205 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:29.205 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:29.205 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.205 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.205 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:29.205 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:29.205 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:29.205 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:29.205 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:29.205 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:29.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949c6a5640 10 monclient: shutdown 2024-05-04T17:45:29.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949c6a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:29.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949c6a5640 1 -- 172.21.15.19:0/809874935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9494072440 msgr2=0x7f949413fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949c6a5640 1 --2- 172.21.15.19:0/809874935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9494072440 0x7f949413fcc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f94880097a0 tx=0x7f948802b8d0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949c6a5640 1 -- 172.21.15.19:0/809874935 shutdown_connections 2024-05-04T17:45:29.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949c6a5640 1 --2- 172.21.15.19:0/809874935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9494072440 0x7f949413fcc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949c6a5640 1 --2- 172.21.15.19:0/809874935 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9494071b20 0x7f9494071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.202+0000 7f949c6a5640 1 -- 172.21.15.19:0/809874935 >> 172.21.15.19:0/809874935 conn(0x7f94940679a0 msgr2=0x7f94940b6530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.203+0000 7f949c6a5640 1 -- 172.21.15.19:0/809874935 shutdown_connections 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.203+0000 7f949c6a5640 1 -- 172.21.15.19:0/809874935 wait complete. 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.203+0000 7f949c6a5640 10 monclient: build_initial_monmap 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.203+0000 7f949c6a5640 10 monclient: monmap: 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:29.209 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:29.210 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.203+0000 7f949c6a5640 1 librados: starting msgr at 2024-05-04T17:45:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.203+0000 7f949c6a5640 1 librados: starting objecter 2024-05-04T17:45:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 1 Processor -- start 2024-05-04T17:45:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 1 -- start start 2024-05-04T17:45:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 1 librados: setting wanted keys 2024-05-04T17:45:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 1 librados: calling monclient init 2024-05-04T17:45:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 10 monclient: init 2024-05-04T17:45:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-04T17:45:29.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9494071b20 0x7f94941d8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 10 monclient(hunting): picked mon.noname-b con 0x7f9494071b20 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9494072440 0x7f94941305f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 10 monclient(hunting): picked mon.noname-c con 0x7f9494072440 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9494140200 0x7f9494130b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 10 monclient(hunting): picked mon.noname-a con 0x7f9494140200 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f949414d670 con 0x7f9494140200 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f949406c670 con 0x7f9494071b20 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f949414d7f0 con 0x7f9494072440 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:29.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.204+0000 7f949c6a5640 10 monclient(hunting): authenticate will time out at 1351.318276s 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949a441640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9494071b20 0x7f94941d8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f9499c40640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9494072440 0x7f94941305f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949ac42640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9494140200 0x7f9494130b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949a441640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9494071b20 0x7f94941d8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36582/0 (socket says 172.21.15.19:36582) 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949a441640 1 -- 172.21.15.19:0/1904169573 learned_addr learned my addr 172.21.15.19:0/1904169573 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949a441640 10 monclient(hunting): get_auth_request con 0x7f9494071b20 auth_method 0 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949a441640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949a441640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949a441640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f9499c40640 10 monclient(hunting): get_auth_request con 0x7f9494072440 auth_method 0 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f9499c40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f9499c40640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f9499c40640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949ac42640 10 monclient(hunting): get_auth_request con 0x7f9494140200 auth_method 0 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949ac42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:29.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949ac42640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:29.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949ac42640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:29.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949a441640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:29.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949a441640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:29.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.205+0000 7f949a441640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:29.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.206+0000 7f9499c40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:29.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.206+0000 7f9499c40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:29.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.206+0000 7f9499c40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:29.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.206+0000 7f949ac42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:29.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.206+0000 7f949ac42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:29.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.206+0000 7f949ac42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:29.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.208+0000 7f949a441640 10 monclient(hunting): handle_auth_done global_id 4426 payload 824 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.208+0000 7f949a441640 1 -- 172.21.15.19:0/1904169573 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9494072440 msgr2=0x7f94941305f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.208+0000 7f949a441640 1 --2- 172.21.15.19:0/1904169573 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9494072440 0x7f94941305f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.208+0000 7f949a441640 1 -- 172.21.15.19:0/1904169573 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9494140200 msgr2=0x7f9494130b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.208+0000 7f949a441640 1 --2- 172.21.15.19:0/1904169573 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9494140200 0x7f9494130b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.208+0000 7f949a441640 10 monclient: _finish_hunting 0 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.208+0000 7f949a441640 1 monclient: found mon.noname-b 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.208+0000 7f949a441640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.208+0000 7f949a441640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.208+0000 7f949a441640 1 -- 172.21.15.19:0/1904169573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9494140950 con 0x7f9494071b20 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.210+0000 7f949a441640 1 --2- 172.21.15.19:0/1904169573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9494071b20 0x7f94941d8100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f948c00e9b0 tx=0x7f948c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.210+0000 7f9492ffd640 10 client.?.objecter ms_handle_connect 0x7f9494071b20 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.210+0000 7f9492ffd640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 1 -- 172.21.15.19:0/1904169573 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c017070 con 0x7f9494071b20 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:29.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 10 monclient: dump: 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 10 monclient: _finish_auth 0 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 10 monclient: _check_auth_tickets 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:29.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 1 -- 172.21.15.19:0/1904169573 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f948c00c9a0 con 0x7f9494071b20 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 1 -- 172.21.15.19:0/1904169573 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c00cd90 con 0x7f9494071b20 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9492ffd640 10 monclient: dump: 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:29.217 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:29.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f949c6a5640 5 monclient: authenticate success, global_id 4426 2024-05-04T17:45:29.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f949c6a5640 10 monclient: _renew_subs 2024-05-04T17:45:29.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f949c6a5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:29.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f949c6a5640 1 -- 172.21.15.19:0/1904169573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94940ae120 con 0x7f9494071b20 2024-05-04T17:45:29.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f9498c3e640 4 set_mon_vals no callback set 2024-05-04T17:45:29.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f949c6a5640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:29.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f949c6a5640 10 monclient: _renew_subs 2024-05-04T17:45:29.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f949c6a5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:29.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f949c6a5640 1 -- 172.21.15.19:0/1904169573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94941d8b20 con 0x7f9494071b20 2024-05-04T17:45:29.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.213+0000 7f949c6a5640 1 librados: init done 2024-05-04T17:45:29.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.214+0000 7f949c6a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:29.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.214+0000 7f949c6a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:29.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.214+0000 7f949c6a5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:29.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.214+0000 7f949c6a5640 1 -- 172.21.15.19:0/1904169573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9494068b10 con 0x7f9494071b20 2024-05-04T17:45:29.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.466+0000 7f9492ffd640 1 -- 172.21.15.19:0/1904169573 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f948c01a070 con 0x7f9494071b20 2024-05-04T17:45:29.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.466+0000 7f9492ffd640 1 --2- 172.21.15.19:0/1904169573 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f948006d7c0 0x7f948006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:29.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.467+0000 7f9499c40640 1 --2- 172.21.15.19:0/1904169573 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f948006d7c0 0x7f948006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:29.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.467+0000 7f9499c40640 10 monclient: get_auth_request con 0x7f948006d7c0 auth_method 0 2024-05-04T17:45:29.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.467+0000 7f9499c40640 1 --2- 172.21.15.19:0/1904169573 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f948006d7c0 0x7f948006fc80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9488036000 tx=0x7f9488008030 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:29.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.467+0000 7f9492ffd640 10 client.4426.objecter ms_handle_connect 0x7f948006d7c0 2024-05-04T17:45:29.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.469+0000 7f9492ffd640 1 -- 172.21.15.19:0/1904169573 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f948c057a40 con 0x7f9494071b20 2024-05-04T17:45:29.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.469+0000 7f9492ffd640 10 client.4426.objecter ms_dispatch 0x7f9494000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:29.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.469+0000 7f9492ffd640 3 client.4426.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:29.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.469+0000 7f9492ffd640 3 client.4426.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:29.470 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.469+0000 7f9492ffd640 20 client.4426.objecter dump_active .. 0 homeless 2024-05-04T17:45:29.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.480+0000 7f9492ffd640 1 -- 172.21.15.19:0/1904169573 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f948c0599a0 con 0x7f9494071b20 2024-05-04T17:45:29.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.480+0000 7f9492ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:29.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.480+0000 7f9492ffd640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:29.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.670+0000 7f949c6a5640 1 -- 172.21.15.19:0/1904169573 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f949406b9f0 con 0x7f948006d7c0 2024-05-04T17:45:29.672 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.671+0000 7f9492ffd640 1 -- 172.21.15.19:0/1904169573 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35442 (secure 0 0 0) 0x7f949406b9f0 con 0x7f948006d7c0 2024-05-04T17:45:29.672 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:45:29.673 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-05-04T17:45:29.674 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 10 librados: watch_flush enter 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 10 librados: watch_flush exit 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 20 client.4426.objecter shutdown clearing up homeless session... 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 10 client.4426.objecter successfully canceled tick 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 -- 172.21.15.19:0/1904169573 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f948006d7c0 msgr2=0x7f948006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 --2- 172.21.15.19:0/1904169573 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f948006d7c0 0x7f948006fc80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9488036000 tx=0x7f9488008030 comp rx=0 tx=0).stop 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 10 monclient: shutdown 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 -- 172.21.15.19:0/1904169573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9494071b20 msgr2=0x7f94941d8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 --2- 172.21.15.19:0/1904169573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9494071b20 0x7f94941d8100 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f948c00e9b0 tx=0x7f948c00ee70 comp rx=0 tx=0).stop 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 -- 172.21.15.19:0/1904169573 shutdown_connections 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 --2- 172.21.15.19:0/1904169573 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f948006d7c0 0x7f948006fc80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 --2- 172.21.15.19:0/1904169573 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9494140200 0x7f9494130b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.675 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 --2- 172.21.15.19:0/1904169573 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9494072440 0x7f94941305f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 --2- 172.21.15.19:0/1904169573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9494071b20 0x7f94941d8100 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 -- 172.21.15.19:0/1904169573 >> 172.21.15.19:0/1904169573 conn(0x7f94940679a0 msgr2=0x7f949413c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:29.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 -- 172.21.15.19:0/1904169573 shutdown_connections 2024-05-04T17:45:29.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 -- 172.21.15.19:0/1904169573 wait complete. 2024-05-04T17:45:29.676 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.673+0000 7f949c6a5640 1 librados: shutdown 2024-05-04T17:45:29.685 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-05-04T17:45:28.838449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218856,"kb_used_data":2672,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755864,"statfs":{"total":773094113280,"available":772870004736,"internally_reserved":0,"allocated":2736128,"data_stored":1482760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.055152"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.457271+0000","last_change":"2024-05-04T17:45:23.672655+0000","last_active":"2024-05-04T17:45:24.457271+0000","last_peered":"2024-05-04T17:45:24.457271+0000","last_clean":"2024-05-04T17:45:24.457271+0000","last_became_active":"2024-05-04T17:45:23.667080+0000","last_became_peered":"2024-05-04T17:45:23.667080+0000","last_unstale":"2024-05-04T17:45:24.457271+0000","last_undegraded":"2024-05-04T17:45:24.457271+0000","last_fullsized":"2024-05-04T17:45:24.457271+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:43:12.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.415067+0000","last_change":"2024-05-04T17:45:23.713140+0000","last_active":"2024-05-04T17:45:24.415067+0000","last_peered":"2024-05-04T17:45:24.415067+0000","last_clean":"2024-05-04T17:45:24.415067+0000","last_became_active":"2024-05-04T17:45:23.686812+0000","last_became_peered":"2024-05-04T17:45:23.686812+0000","last_unstale":"2024-05-04T17:45:24.415067+0000","last_undegraded":"2024-05-04T17:45:24.415067+0000","last_fullsized":"2024-05-04T17:45:24.415067+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:56:49.147299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.443406+0000","last_change":"2024-05-04T17:45:23.379564+0000","last_active":"2024-05-04T17:45:24.443406+0000","last_peered":"2024-05-04T17:45:24.443406+0000","last_clean":"2024-05-04T17:45:24.443406+0000","last_became_active":"2024-05-04T17:45:23.370117+0000","last_became_peered":"2024-05-04T17:45:23.370117+0000","last_unstale":"2024-05-04T17:45:24.443406+0000","last_undegraded":"2024-05-04T17:45:24.443406+0000","last_fullsized":"2024-05-04T17:45:24.443406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:39:45.488747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.907211+0000","last_change":"2024-05-04T17:45:23.488141+0000","last_active":"2024-05-04T17:45:24.907211+0000","last_peered":"2024-05-04T17:45:24.907211+0000","last_clean":"2024-05-04T17:45:24.907211+0000","last_became_active":"2024-05-04T17:45:23.481564+0000","last_became_peered":"2024-05-04T17:45:23.481564+0000","last_unstale":"2024-05-04T17:45:24.907211+0000","last_undegraded":"2024-05-04T17:45:24.907211+0000","last_fullsized":"2024-05-04T17:45:24.907211+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:23:41.989802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.446577+0000","last_change":"2024-05-04T17:45:23.491691+0000","last_active":"2024-05-04T17:45:24.446577+0000","last_peered":"2024-05-04T17:45:24.446577+0000","last_clean":"2024-05-04T17:45:24.446577+0000","last_became_active":"2024-05-04T17:45:23.480170+0000","last_became_peered":"2024-05-04T17:45:23.480170+0000","last_unstale":"2024-05-04T17:45:24.446577+0000","last_undegraded":"2024-05-04T17:45:24.446577+0000","last_fullsized":"2024-05-04T17:45:24.446577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:18:07.981076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.543676+0000","last_change":"2024-05-04T17:45:23.301096+0000","last_active":"2024-05-04T17:45:24.543676+0000","last_peered":"2024-05-04T17:45:24.543676+0000","last_clean":"2024-05-04T17:45:24.543676+0000","last_became_active":"2024-05-04T17:45:23.265024+0000","last_became_peered":"2024-05-04T17:45:23.265024+0000","last_unstale":"2024-05-04T17:45:24.543676+0000","last_undegraded":"2024-05-04T17:45:24.543676+0000","last_fullsized":"2024-05-04T17:45:24.543676+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:48:10.154363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.430779+0000","last_change":"2024-05-04T17:45:23.292542+0000","last_active":"2024-05-04T17:45:24.430779+0000","last_peered":"2024-05-04T17:45:24.430779+0000","last_clean":"2024-05-04T17:45:24.430779+0000","last_became_active":"2024-05-04T17:45:23.262086+0000","last_became_peered":"2024-05-04T17:45:23.262086+0000","last_unstale":"2024-05-04T17:45:24.430779+0000","last_undegraded":"2024-05-04T17:45:24.430779+0000","last_fullsized":"2024-05-04T17:45:24.430779+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:16:49.542097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.440169+0000","last_change":"2024-05-04T17:45:23.593473+0000","last_active":"2024-05-04T17:45:24.440169+0000","last_peered":"2024-05-04T17:45:24.440169+0000","last_clean":"2024-05-04T17:45:24.440169+0000","last_became_active":"2024-05-04T17:45:23.583946+0000","last_became_peered":"2024-05-04T17:45:23.583946+0000","last_unstale":"2024-05-04T17:45:24.440169+0000","last_undegraded":"2024-05-04T17:45:24.440169+0000","last_fullsized":"2024-05-04T17:45:24.440169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:19:20.455748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'76","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.897067+0000","last_change":"2024-05-04T17:45:18.398964+0000","last_active":"2024-05-04T17:45:24.897067+0000","last_peered":"2024-05-04T17:45:24.897067+0000","last_clean":"2024-05-04T17:45:24.897067+0000","last_became_active":"2024-05-04T17:45:17.720848+0000","last_became_peered":"2024-05-04T17:45:17.720848+0000","last_unstale":"2024-05-04T17:45:24.897067+0000","last_undegraded":"2024-05-04T17:45:24.897067+0000","last_fullsized":"2024-05-04T17:45:24.897067+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:03.919442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:03.919442+0000","last_clean_scrub_stamp":"2024-05-04T17:45:03.919442+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:32:22.959179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":483725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":20,"seq":85899345929,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":85885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378633,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":85885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378634,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":85885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":16,"seq":68719476744,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":85885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":85885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":85885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":483725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T17:45:29.686 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T17:45:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.784+0000 7f6eda591640 10 monclient: get_monmap_and_config 2024-05-04T17:45:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.784+0000 7f6eda591640 10 monclient: build_initial_monmap 2024-05-04T17:45:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.784+0000 7f6eda591640 10 monclient: monmap: 2024-05-04T17:45:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:29.784980+0000 2024-05-04T17:45:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:29.784980+0000 2024-05-04T17:45:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:29.785 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:29.786 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:29.786 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:29.786 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:29.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.784+0000 7f6eda591640 1 Processor -- start 2024-05-04T17:45:29.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 1 -- start start 2024-05-04T17:45:29.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 10 monclient: init 2024-05-04T17:45:29.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6ed40a9720 0x7f6ed40aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 10 monclient(hunting): picked mon.noname-a con 0x7f6ed40a9720 addr v2:172.21.15.19:3300/0 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ed40a9b00 addr v1:172.21.15.186:6789/0 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ed40b2e90 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ed40abc40 con 0x7f6ed40b2e90 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ed4146fa0 con 0x7f6ed40a9b00 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f6ed4146090 con 0x7f6ed40a9720 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.785+0000 7f6eda591640 10 monclient(hunting): authenticate will time out at 1351.899312s 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.786+0000 7f6ed3fff640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f6ed40a9720 0x7f6ed40aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.786+0000 7f6ed37fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6ed40a9b00 0x7f6ed4146890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:46532/0 (socket says 172.21.15.19:46532) 2024-05-04T17:45:29.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.786+0000 7f6ed37fe640 1 -- 172.21.15.19:0/34734642 learned_addr learned my addr 172.21.15.19:0/34734642 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:29.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.786+0000 7f6ed3fff640 10 monclient(hunting): get_auth_request con 0x7f6ed40a9720 auth_method 0 2024-05-04T17:45:29.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.786+0000 7f6ed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:29.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.786+0000 7f6ed3fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:29.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.786+0000 7f6ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.787+0000 7f6ed3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.787+0000 7f6ed3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.787+0000 7f6ed3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.788+0000 7f6ed2ffd640 1 -- 172.21.15.19:0/34734642 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1001624480 0 0) 0x7f6ed4146fa0 con 0x7f6ed40a9b00 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.788+0000 7f6ed2ffd640 10 monclient(hunting): _negotiate 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.788+0000 7f6ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.788+0000 7f6ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.788+0000 7f6ed2ffd640 10 monclient(hunting): my global_id is 4429 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.788+0000 7f6ed2ffd640 1 -- 172.21.15.19:0/34734642 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ebc003520 con 0x7f6ed40a9b00 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.789+0000 7f6ed3fff640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.789+0000 7f6ed3fff640 1 -- 172.21.15.19:0/34734642 >> v1:172.21.15.186:6789/0 conn(0x7f6ed40a9b00 legacy=0x7f6ed4146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.789+0000 7f6ed3fff640 1 -- 172.21.15.19:0/34734642 >> v1:172.21.15.19:6789/0 conn(0x7f6ed40b2e90 legacy=0x7f6ed413fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.789+0000 7f6ed3fff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:29.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.789+0000 7f6ed3fff640 1 monclient: found mon.noname-a 2024-05-04T17:45:29.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.789+0000 7f6ed3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:29.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.789+0000 7f6ed3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:45:29.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.789+0000 7f6ed3fff640 1 -- 172.21.15.19:0/34734642 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed4146370 con 0x7f6ed40a9720 2024-05-04T17:45:29.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.792+0000 7f6ed3fff640 1 --2- 172.21.15.19:0/34734642 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ed40a9720 0x7f6ed40aa890 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6ec40097d0 tx=0x7f6ec402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1015ba6bc4e6a90 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:29.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 1 -- 172.21.15.19:0/34734642 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4034070 con 0x7f6ed40a9720 2024-05-04T17:45:29.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:29.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:45:29.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 10 monclient: dump: 2024-05-04T17:45:29.797 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:29.797 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:29.797 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.797 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.797 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:29.797 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 10 monclient: _finish_auth 0 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 10 monclient: _check_auth_tickets 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 1 -- 172.21.15.19:0/34734642 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec4030610 con 0x7f6ed40a9720 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 1 -- 172.21.15.19:0/34734642 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec40309e0 con 0x7f6ed40a9720 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6ed2ffd640 10 monclient: dump: 2024-05-04T17:45:29.798 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:29.799 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:29.799 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.799 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.799 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:29.799 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:29.799 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:29.799 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:29.799 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:29.799 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:29.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6eda591640 5 monclient: authenticate success, global_id 4347 2024-05-04T17:45:29.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6eda591640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:29.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.796+0000 7f6eda591640 4 set_mon_vals no callback set 2024-05-04T17:45:29.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.797+0000 7f6eda591640 10 monclient: shutdown 2024-05-04T17:45:29.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.797+0000 7f6eda591640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:29.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.797+0000 7f6eda591640 1 -- 172.21.15.19:0/34734642 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ed40a9720 msgr2=0x7f6ed40aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.797+0000 7f6eda591640 1 --2- 172.21.15.19:0/34734642 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ed40a9720 0x7f6ed40aa890 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6ec40097d0 tx=0x7f6ec402b960 comp rx=0 tx=0).stop 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.797+0000 7f6eda591640 1 -- 172.21.15.19:0/34734642 shutdown_connections 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.797+0000 7f6eda591640 1 --2- 172.21.15.19:0/34734642 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ed40a9720 0x7f6ed40aa890 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.797+0000 7f6eda591640 1 -- 172.21.15.19:0/34734642 >> 172.21.15.19:0/34734642 conn(0x7f6ed4067780 msgr2=0x7f6ed40b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.797+0000 7f6eda591640 1 -- 172.21.15.19:0/34734642 shutdown_connections 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.797+0000 7f6eda591640 1 -- 172.21.15.19:0/34734642 wait complete. 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.798+0000 7f6eda591640 10 monclient: build_initial_monmap 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.798+0000 7f6eda591640 10 monclient: monmap: 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:29.803 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.798+0000 7f6eda591640 1 librados: starting msgr at 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.798+0000 7f6eda591640 1 librados: starting objecter 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.798+0000 7f6eda591640 1 Processor -- start 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.798+0000 7f6eda591640 1 -- start start 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.798+0000 7f6eda591640 1 librados: setting wanted keys 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.798+0000 7f6eda591640 1 librados: calling monclient init 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.798+0000 7f6eda591640 10 monclient: init 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed40a9720 0x7f6ed41436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 10 monclient(hunting): picked mon.noname-b con 0x7f6ed40a9720 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:29.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6ed40a9b00 0x7f6ed4140cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 10 monclient(hunting): picked mon.noname-c con 0x7f6ed40a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ed40b2e90 0x7f6ed41411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 10 monclient(hunting): picked mon.noname-a con 0x7f6ed40b2e90 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed414e430 con 0x7f6ed40b2e90 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed414e5b0 con 0x7f6ed40a9720 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f6ed414e730 con 0x7f6ed40a9b00 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6eda591640 10 monclient(hunting): authenticate will time out at 1351.913079s 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6ed3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed40a9720 0x7f6ed41436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.799+0000 7f6ed37fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6ed40a9b00 0x7f6ed4140cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed8b2e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ed40b2e90 0x7f6ed41411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:29.805 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed40a9720 0x7f6ed41436a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36594/0 (socket says 172.21.15.19:36594) 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed3fff640 1 -- 172.21.15.19:0/3444851134 learned_addr learned my addr 172.21.15.19:0/3444851134 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed3fff640 10 monclient(hunting): get_auth_request con 0x7f6ed40a9720 auth_method 0 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed3fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed37fe640 10 monclient(hunting): get_auth_request con 0x7f6ed40a9b00 auth_method 0 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed37fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed8b2e640 10 monclient(hunting): get_auth_request con 0x7f6ed40b2e90 auth_method 0 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed8b2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed8b2e640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed8b2e640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:29.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:29.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:29.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:29.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:29.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:29.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed8b2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:29.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed8b2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:29.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.800+0000 7f6ed8b2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.803+0000 7f6ed3fff640 10 monclient(hunting): handle_auth_done global_id 4432 payload 824 2024-05-04T17:45:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.803+0000 7f6ed3fff640 1 -- 172.21.15.19:0/3444851134 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6ed40a9b00 msgr2=0x7f6ed4140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.803+0000 7f6ed3fff640 1 --2- 172.21.15.19:0/3444851134 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6ed40a9b00 0x7f6ed4140cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.803+0000 7f6ed3fff640 1 -- 172.21.15.19:0/3444851134 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ed40b2e90 msgr2=0x7f6ed41411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.803+0000 7f6ed3fff640 1 --2- 172.21.15.19:0/3444851134 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ed40b2e90 0x7f6ed41411f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.803+0000 7f6ed3fff640 10 monclient: _finish_hunting 0 2024-05-04T17:45:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.803+0000 7f6ed3fff640 1 monclient: found mon.noname-b 2024-05-04T17:45:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.803+0000 7f6ed3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.803+0000 7f6ed3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:29.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.803+0000 7f6ed3fff640 1 -- 172.21.15.19:0/3444851134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed41354f0 con 0x7f6ed40a9720 2024-05-04T17:45:29.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.805+0000 7f6ed3fff640 1 --2- 172.21.15.19:0/3444851134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed40a9720 0x7f6ed41436a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6ec402bf70 tx=0x7f6ec40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:29.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.805+0000 7f6ed0ff9640 10 client.?.objecter ms_handle_connect 0x7f6ed40a9720 2024-05-04T17:45:29.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.805+0000 7f6ed0ff9640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:29.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6ed0ff9640 1 -- 172.21.15.19:0/3444851134 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4034040 con 0x7f6ed40a9720 2024-05-04T17:45:29.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:29.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6ed0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:29.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6ed0ff9640 10 monclient: dump: 2024-05-04T17:45:29.810 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:29.810 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:29.810 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.810 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:29.810 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:29.810 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:29.810 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6ed0ff9640 10 monclient: _finish_auth 0 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6ed0ff9640 10 monclient: _check_auth_tickets 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6ed0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6eda591640 5 monclient: authenticate success, global_id 4432 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6eda591640 10 monclient: _renew_subs 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6eda591640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6eda591640 1 -- 172.21.15.19:0/3444851134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed4136950 con 0x7f6ed40a9720 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6eda591640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6eda591640 10 monclient: _renew_subs 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6eda591640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:29.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6eda591640 1 -- 172.21.15.19:0/3444851134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed4136f10 con 0x7f6ed40a9720 2024-05-04T17:45:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.808+0000 7f6eda591640 1 librados: init done 2024-05-04T17:45:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.809+0000 7f6eda591640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.809+0000 7f6eda591640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.809+0000 7f6eda591640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:29.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:29.809+0000 7f6eda591640 1 -- 172.21.15.19:0/3444851134 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed406a700 con 0x7f6ed40a9720 2024-05-04T17:45:30.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.059+0000 7f6ed0ff9640 1 -- 172.21.15.19:0/3444851134 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec40304d0 con 0x7f6ed40a9720 2024-05-04T17:45:30.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.059+0000 7f6ed0ff9640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:30.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.059+0000 7f6ed0ff9640 1 -- 172.21.15.19:0/3444851134 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4039b90 con 0x7f6ed40a9720 2024-05-04T17:45:30.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.059+0000 7f6ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:30.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.059+0000 7f6ed0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:30.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.059+0000 7f6ed27fc640 4 set_mon_vals no callback set 2024-05-04T17:45:30.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.059+0000 7f6ed0ff9640 10 monclient: dump: 2024-05-04T17:45:30.063 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:30.063 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:30.063 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:30.063 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:30.063 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:30.063 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:30.063 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:30.064 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:30.064 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:30.064 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:30.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.059+0000 7f6ed0ff9640 1 -- 172.21.15.19:0/3444851134 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6ec4037070 con 0x7f6ed40a9720 2024-05-04T17:45:30.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.060+0000 7f6ed0ff9640 1 --2- 172.21.15.19:0/3444851134 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f6ec006d7f0 0x7f6ec006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:30.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.060+0000 7f6ed37fe640 1 --2- 172.21.15.19:0/3444851134 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f6ec006d7f0 0x7f6ec006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:30.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.060+0000 7f6ed37fe640 10 monclient: get_auth_request con 0x7f6ec006d7f0 auth_method 0 2024-05-04T17:45:30.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.061+0000 7f6ed37fe640 1 --2- 172.21.15.19:0/3444851134 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f6ec006d7f0 0x7f6ec006fcb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4142570 tx=0x7f6ec8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:30.064 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.061+0000 7f6ed0ff9640 10 client.4432.objecter ms_handle_connect 0x7f6ec006d7f0 2024-05-04T17:45:30.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.063+0000 7f6ed0ff9640 1 -- 172.21.15.19:0/3444851134 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f6ec4079bb0 con 0x7f6ed40a9720 2024-05-04T17:45:30.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.063+0000 7f6ed0ff9640 10 client.4432.objecter ms_dispatch 0x7f6ed4000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:30.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.063+0000 7f6ed0ff9640 3 client.4432.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:30.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.063+0000 7f6ed0ff9640 3 client.4432.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:30.065 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.063+0000 7f6ed0ff9640 20 client.4432.objecter dump_active .. 0 homeless 2024-05-04T17:45:30.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.074+0000 7f6ed0ff9640 1 -- 172.21.15.19:0/3444851134 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6ed41354f0 con 0x7f6ed40a9720 2024-05-04T17:45:30.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.074+0000 7f6ed0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:30.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.074+0000 7f6ed0ff9640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:30.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.267+0000 7f6eda591640 1 -- 172.21.15.19:0/3444851134 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6ed413ae70 con 0x7f6ec006d7f0 2024-05-04T17:45:30.269 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.268+0000 7f6ed0ff9640 1 -- 172.21.15.19:0/3444851134 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35442 (secure 0 0 0) 0x7f6ed413ae70 con 0x7f6ec006d7f0 2024-05-04T17:45:30.269 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:45:30.270 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 10 librados: watch_flush enter 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 10 librados: watch_flush exit 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 20 client.4432.objecter shutdown clearing up homeless session... 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 10 client.4432.objecter successfully canceled tick 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 1 -- 172.21.15.19:0/3444851134 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f6ec006d7f0 msgr2=0x7f6ec006fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 1 --2- 172.21.15.19:0/3444851134 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f6ec006d7f0 0x7f6ec006fcb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4142570 tx=0x7f6ec8009040 comp rx=0 tx=0).stop 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 10 monclient: shutdown 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 1 -- 172.21.15.19:0/3444851134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed40a9720 msgr2=0x7f6ed41436a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 1 --2- 172.21.15.19:0/3444851134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed40a9720 0x7f6ed41436a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6ec402bf70 tx=0x7f6ec40046c0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 1 -- 172.21.15.19:0/3444851134 shutdown_connections 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 1 --2- 172.21.15.19:0/3444851134 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f6ec006d7f0 0x7f6ec006fcb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 1 --2- 172.21.15.19:0/3444851134 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f6ed40b2e90 0x7f6ed41411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 1 --2- 172.21.15.19:0/3444851134 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f6ed40a9b00 0x7f6ed4140cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 1 --2- 172.21.15.19:0/3444851134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed40a9720 0x7f6ed41436a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.270+0000 7f6eda591640 1 -- 172.21.15.19:0/3444851134 >> 172.21.15.19:0/3444851134 conn(0x7f6ed4067780 msgr2=0x7f6ed413fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:30.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.271+0000 7f6eda591640 1 -- 172.21.15.19:0/3444851134 shutdown_connections 2024-05-04T17:45:30.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.271+0000 7f6eda591640 1 -- 172.21.15.19:0/3444851134 wait complete. 2024-05-04T17:45:30.273 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.271+0000 7f6eda591640 1 librados: shutdown 2024-05-04T17:45:30.282 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-05-04T17:45:28.838449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218856,"kb_used_data":2672,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755864,"statfs":{"total":773094113280,"available":772870004736,"internally_reserved":0,"allocated":2736128,"data_stored":1482760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.055152"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.457271+0000","last_change":"2024-05-04T17:45:23.672655+0000","last_active":"2024-05-04T17:45:24.457271+0000","last_peered":"2024-05-04T17:45:24.457271+0000","last_clean":"2024-05-04T17:45:24.457271+0000","last_became_active":"2024-05-04T17:45:23.667080+0000","last_became_peered":"2024-05-04T17:45:23.667080+0000","last_unstale":"2024-05-04T17:45:24.457271+0000","last_undegraded":"2024-05-04T17:45:24.457271+0000","last_fullsized":"2024-05-04T17:45:24.457271+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:43:12.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.415067+0000","last_change":"2024-05-04T17:45:23.713140+0000","last_active":"2024-05-04T17:45:24.415067+0000","last_peered":"2024-05-04T17:45:24.415067+0000","last_clean":"2024-05-04T17:45:24.415067+0000","last_became_active":"2024-05-04T17:45:23.686812+0000","last_became_peered":"2024-05-04T17:45:23.686812+0000","last_unstale":"2024-05-04T17:45:24.415067+0000","last_undegraded":"2024-05-04T17:45:24.415067+0000","last_fullsized":"2024-05-04T17:45:24.415067+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:56:49.147299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.443406+0000","last_change":"2024-05-04T17:45:23.379564+0000","last_active":"2024-05-04T17:45:24.443406+0000","last_peered":"2024-05-04T17:45:24.443406+0000","last_clean":"2024-05-04T17:45:24.443406+0000","last_became_active":"2024-05-04T17:45:23.370117+0000","last_became_peered":"2024-05-04T17:45:23.370117+0000","last_unstale":"2024-05-04T17:45:24.443406+0000","last_undegraded":"2024-05-04T17:45:24.443406+0000","last_fullsized":"2024-05-04T17:45:24.443406+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:39:45.488747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.907211+0000","last_change":"2024-05-04T17:45:23.488141+0000","last_active":"2024-05-04T17:45:24.907211+0000","last_peered":"2024-05-04T17:45:24.907211+0000","last_clean":"2024-05-04T17:45:24.907211+0000","last_became_active":"2024-05-04T17:45:23.481564+0000","last_became_peered":"2024-05-04T17:45:23.481564+0000","last_unstale":"2024-05-04T17:45:24.907211+0000","last_undegraded":"2024-05-04T17:45:24.907211+0000","last_fullsized":"2024-05-04T17:45:24.907211+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:23:41.989802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.446577+0000","last_change":"2024-05-04T17:45:23.491691+0000","last_active":"2024-05-04T17:45:24.446577+0000","last_peered":"2024-05-04T17:45:24.446577+0000","last_clean":"2024-05-04T17:45:24.446577+0000","last_became_active":"2024-05-04T17:45:23.480170+0000","last_became_peered":"2024-05-04T17:45:23.480170+0000","last_unstale":"2024-05-04T17:45:24.446577+0000","last_undegraded":"2024-05-04T17:45:24.446577+0000","last_fullsized":"2024-05-04T17:45:24.446577+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:18:07.981076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.543676+0000","last_change":"2024-05-04T17:45:23.301096+0000","last_active":"2024-05-04T17:45:24.543676+0000","last_peered":"2024-05-04T17:45:24.543676+0000","last_clean":"2024-05-04T17:45:24.543676+0000","last_became_active":"2024-05-04T17:45:23.265024+0000","last_became_peered":"2024-05-04T17:45:23.265024+0000","last_unstale":"2024-05-04T17:45:24.543676+0000","last_undegraded":"2024-05-04T17:45:24.543676+0000","last_fullsized":"2024-05-04T17:45:24.543676+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:48:10.154363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.430779+0000","last_change":"2024-05-04T17:45:23.292542+0000","last_active":"2024-05-04T17:45:24.430779+0000","last_peered":"2024-05-04T17:45:24.430779+0000","last_clean":"2024-05-04T17:45:24.430779+0000","last_became_active":"2024-05-04T17:45:23.262086+0000","last_became_peered":"2024-05-04T17:45:23.262086+0000","last_unstale":"2024-05-04T17:45:24.430779+0000","last_undegraded":"2024-05-04T17:45:24.430779+0000","last_fullsized":"2024-05-04T17:45:24.430779+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:16:49.542097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.440169+0000","last_change":"2024-05-04T17:45:23.593473+0000","last_active":"2024-05-04T17:45:24.440169+0000","last_peered":"2024-05-04T17:45:24.440169+0000","last_clean":"2024-05-04T17:45:24.440169+0000","last_became_active":"2024-05-04T17:45:23.583946+0000","last_became_peered":"2024-05-04T17:45:23.583946+0000","last_unstale":"2024-05-04T17:45:24.440169+0000","last_undegraded":"2024-05-04T17:45:24.440169+0000","last_fullsized":"2024-05-04T17:45:24.440169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:19:20.455748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'76","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-05-04T17:45:24.897067+0000","last_change":"2024-05-04T17:45:18.398964+0000","last_active":"2024-05-04T17:45:24.897067+0000","last_peered":"2024-05-04T17:45:24.897067+0000","last_clean":"2024-05-04T17:45:24.897067+0000","last_became_active":"2024-05-04T17:45:17.720848+0000","last_became_peered":"2024-05-04T17:45:17.720848+0000","last_unstale":"2024-05-04T17:45:24.897067+0000","last_undegraded":"2024-05-04T17:45:24.897067+0000","last_fullsized":"2024-05-04T17:45:24.897067+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:03.919442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:03.919442+0000","last_clean_scrub_stamp":"2024-05-04T17:45:03.919442+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:32:22.959179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":483725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":20,"seq":85899345929,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":85885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":19,"seq":81604378633,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":85885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604378634,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":85885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":16,"seq":68719476744,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":241664,"data_stored":85885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542151,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":85885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":85885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":483725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T17:45:30.283 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-04T17:45:30.283 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-04T17:45:30.283 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-04T17:45:30.283 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-04T17:45:30.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.381+0000 7f72d7045640 10 monclient: get_monmap_and_config 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.381+0000 7f72d7045640 10 monclient: build_initial_monmap 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.381+0000 7f72d7045640 10 monclient: monmap: 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:45:30.381928+0000 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:45:30.381928+0000 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T17:45:30.382 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:30.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.381+0000 7f72d7045640 1 Processor -- start 2024-05-04T17:45:30.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 1 -- start start 2024-05-04T17:45:30.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 10 monclient: init 2024-05-04T17:45:30.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:30.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-05-04T17:45:30.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f72d00b8f30 0x7f72d013fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:30.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 10 monclient(hunting): picked mon.noname-b con 0x7f72d00b8f30 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:30.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f72d00aa8b0 0x7f72d01403e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:30.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 10 monclient(hunting): picked mon.noname-c con 0x7f72d00aa8b0 addr v2:172.21.15.186:3300/0 2024-05-04T17:45:30.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72d0146050 addr v1:172.21.15.19:6789/0 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72d00a9d40 con 0x7f72d0146050 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f72d00b9310 con 0x7f72d00aa8b0 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.382+0000 7f72d7045640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f72d00a98b0 con 0x7f72d00b8f30 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.383+0000 7f72d7045640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.383+0000 7f72d7045640 10 monclient(hunting): authenticate will time out at 1352.496660s 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.383+0000 7f72cffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f72d00aa8b0 0x7f72d01403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.383+0000 7f72d4de1640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f72d00b8f30 0x7f72d013fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.383+0000 7f72d55e2640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f72d0146050 0x7f72d01464e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:42428/0 (socket says 172.21.15.19:42428) 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.383+0000 7f72d55e2640 1 -- 172.21.15.19:0/2345506414 learned_addr learned my addr 172.21.15.19:0/2345506414 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.383+0000 7f72cffff640 1 --2- 172.21.15.19:0/2345506414 >> v2:172.21.15.186:3300/0 conn(0x7f72d00aa8b0 0x7f72d01403e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36606/0 (socket says 172.21.15.19:36606) 2024-05-04T17:45:30.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.383+0000 7f72cffff640 10 monclient(hunting): get_auth_request con 0x7f72d00aa8b0 auth_method 0 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.383+0000 7f72cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.383+0000 7f72cffff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.383+0000 7f72cffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.384+0000 7f72d4de1640 10 monclient(hunting): get_auth_request con 0x7f72d00b8f30 auth_method 0 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.384+0000 7f72d4de1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.384+0000 7f72d4de1640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.384+0000 7f72d4de1640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.384+0000 7f72cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.384+0000 7f72cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.384+0000 7f72cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.384+0000 7f72d4de1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.384+0000 7f72d4de1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:30.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.384+0000 7f72d4de1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:30.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.386+0000 7f72d4de1640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-05-04T17:45:30.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.386+0000 7f72d4de1640 1 -- 172.21.15.19:0/2345506414 >> v2:172.21.15.186:3300/0 conn(0x7f72d00aa8b0 msgr2=0x7f72d01403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:45:30.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.386+0000 7f72d4de1640 1 --2- 172.21.15.19:0/2345506414 >> v2:172.21.15.186:3300/0 conn(0x7f72d00aa8b0 0x7f72d01403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.386+0000 7f72d4de1640 1 -- 172.21.15.19:0/2345506414 >> v1:172.21.15.19:6789/0 conn(0x7f72d0146050 legacy=0x7f72d01464e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:30.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.386+0000 7f72d4de1640 10 monclient: _finish_hunting 0 2024-05-04T17:45:30.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.386+0000 7f72d4de1640 1 monclient: found mon.noname-b 2024-05-04T17:45:30.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.386+0000 7f72d4de1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:30.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.386+0000 7f72d4de1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T17:45:30.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.386+0000 7f72d4de1640 1 -- 172.21.15.19:0/2345506414 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d00abbf0 con 0x7f72d00b8f30 2024-05-04T17:45:30.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.388+0000 7f72d4de1640 1 --2- 172.21.15.19:0/2345506414 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f72d00b8f30 0x7f72d013fea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f72c40097a0 tx=0x7f72c402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce2ba903f01f4b20 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:30.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 1 -- 172.21.15.19:0/2345506414 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c40315a0 con 0x7f72d00b8f30 2024-05-04T17:45:30.393 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 10 monclient: dump: 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 10 monclient: _finish_auth 0 2024-05-04T17:45:30.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 10 monclient: _check_auth_tickets 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 1 -- 172.21.15.19:0/2345506414 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c4037070 con 0x7f72d00b8f30 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72d7045640 5 monclient: authenticate success, global_id 4322 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72d7045640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72d7045640 10 monclient: get_monmap_and_config success 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72d7045640 4 set_mon_vals no callback set 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 1 -- 172.21.15.19:0/2345506414 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4031960 con 0x7f72d00b8f30 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.393+0000 7f72cf7fe640 10 monclient: dump: 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:30.395 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:30.396 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:30.396 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:30.396 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:30.396 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:30.396 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:30.396 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:30.396 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:30.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.394+0000 7f72d7045640 10 monclient: shutdown 2024-05-04T17:45:30.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.394+0000 7f72d7045640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:30.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.394+0000 7f72d7045640 1 -- 172.21.15.19:0/2345506414 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f72d00b8f30 msgr2=0x7f72d013fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:30.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.394+0000 7f72d7045640 1 --2- 172.21.15.19:0/2345506414 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f72d00b8f30 0x7f72d013fea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f72c40097a0 tx=0x7f72c402b840 comp rx=0 tx=0).stop 2024-05-04T17:45:30.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.394+0000 7f72d7045640 1 -- 172.21.15.19:0/2345506414 shutdown_connections 2024-05-04T17:45:30.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.394+0000 7f72d7045640 1 --2- 172.21.15.19:0/2345506414 >> v2:172.21.15.186:3300/0 conn(0x7f72d00aa8b0 0x7f72d01403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.394+0000 7f72d7045640 1 --2- 172.21.15.19:0/2345506414 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f72d00b8f30 0x7f72d013fea0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.399 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.394+0000 7f72d7045640 1 -- 172.21.15.19:0/2345506414 >> 172.21.15.19:0/2345506414 conn(0x7f72d00676f0 msgr2=0x7f72d00b8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.394+0000 7f72d7045640 1 -- 172.21.15.19:0/2345506414 shutdown_connections 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.394+0000 7f72d7045640 1 -- 172.21.15.19:0/2345506414 wait complete. 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.394+0000 7f72d7045640 10 monclient: build_initial_monmap 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.394+0000 7f72d7045640 10 monclient: monmap: 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:30.400 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.395+0000 7f72d7045640 1 librados: starting msgr at 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.395+0000 7f72d7045640 1 librados: starting objecter 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.395+0000 7f72d7045640 1 Processor -- start 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.395+0000 7f72d7045640 1 -- start start 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.395+0000 7f72d7045640 1 librados: setting wanted keys 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.395+0000 7f72d7045640 1 librados: calling monclient init 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.395+0000 7f72d7045640 10 monclient: init 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 10 monclient: _reopen_session rank -1 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f72d00aa8b0 0x7f72d00adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 10 monclient(hunting): picked mon.noname-a con 0x7f72d00aa8b0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f72d00b8f30 0x7f72d00ac340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 10 monclient(hunting): picked mon.noname-c con 0x7f72d00b8f30 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72d0146050 0x7f72d00b7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:30.401 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 10 monclient(hunting): picked mon.noname-b con 0x7f72d0146050 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f72d014d3c0 con 0x7f72d00aa8b0 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72d00a94b0 con 0x7f72d0146050 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 10 monclient(hunting): start opening mon connection 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f72d014d540 con 0x7f72d00b8f30 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 10 monclient(hunting): _renew_subs 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d7045640 10 monclient(hunting): authenticate will time out at 1352.509765s 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72cffff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f72d00b8f30 0x7f72d00ac340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d55e2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72d0146050 0x7f72d00b7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d4de1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f72d00aa8b0 0x7f72d00adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d55e2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72d0146050 0x7f72d00b7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:36612/0 (socket says 172.21.15.19:36612) 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d55e2640 1 -- 172.21.15.19:0/846475836 learned_addr learned my addr 172.21.15.19:0/846475836 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:45:30.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d55e2640 10 monclient(hunting): get_auth_request con 0x7f72d0146050 auth_method 0 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d55e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d55e2640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72d55e2640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.396+0000 7f72cffff640 10 monclient(hunting): get_auth_request con 0x7f72d00b8f30 auth_method 0 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72cffff640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72cffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72d4de1640 10 monclient(hunting): get_auth_request con 0x7f72d00aa8b0 auth_method 0 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72d4de1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72d4de1640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72d4de1640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72d55e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72d55e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72d55e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:30.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:30.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:30.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72d4de1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:45:30.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72d4de1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:45:30.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.397+0000 7f72d4de1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:45:30.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.400+0000 7f72d55e2640 10 monclient(hunting): handle_auth_done global_id 4438 payload 824 2024-05-04T17:45:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.400+0000 7f72d55e2640 1 -- 172.21.15.19:0/846475836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f72d00b8f30 msgr2=0x7f72d00ac340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.400+0000 7f72d55e2640 1 --2- 172.21.15.19:0/846475836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f72d00b8f30 0x7f72d00ac340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.400+0000 7f72d55e2640 1 -- 172.21.15.19:0/846475836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f72d00aa8b0 msgr2=0x7f72d00adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.400+0000 7f72d55e2640 1 --2- 172.21.15.19:0/846475836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f72d00aa8b0 0x7f72d00adca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.400+0000 7f72d55e2640 10 monclient: _finish_hunting 0 2024-05-04T17:45:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.400+0000 7f72d55e2640 1 monclient: found mon.noname-b 2024-05-04T17:45:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.400+0000 7f72d55e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:45:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.400+0000 7f72d55e2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:45:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.400+0000 7f72d55e2640 1 -- 172.21.15.19:0/846475836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d00ad330 con 0x7f72d0146050 2024-05-04T17:45:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.402+0000 7f72d55e2640 1 --2- 172.21.15.19:0/846475836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72d0146050 0x7f72d00b7680 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f72c800c9f0 tx=0x7f72c800ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.402+0000 7f72cd7fa640 10 client.?.objecter ms_handle_connect 0x7f72d0146050 2024-05-04T17:45:30.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.402+0000 7f72cd7fa640 10 client.?.objecter resend_mon_ops 2024-05-04T17:45:30.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 1 -- 172.21.15.19:0/846475836 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8017070 con 0x7f72d0146050 2024-05-04T17:45:30.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:30.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:45:30.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 10 monclient: dump: 2024-05-04T17:45:30.408 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:30.408 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 10 monclient: _finish_auth 0 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 10 monclient: _check_auth_tickets 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 1 -- 172.21.15.19:0/846475836 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c800ab70 con 0x7f72d0146050 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72d7045640 5 monclient: authenticate success, global_id 4438 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72d7045640 10 monclient: _renew_subs 2024-05-04T17:45:30.409 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72d7045640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72d7045640 1 -- 172.21.15.19:0/846475836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d00b7bc0 con 0x7f72d0146050 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 10 monclient: handle_config config(0 keys) 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 1 -- 172.21.15.19:0/846475836 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8012400 con 0x7f72d0146050 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72cd7fa640 10 monclient: dump: 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:45:30.410 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:45:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:45:30.411 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T17:45:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.406+0000 7f72ceffd640 4 set_mon_vals no callback set 2024-05-04T17:45:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.407+0000 7f72d7045640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:45:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.407+0000 7f72d7045640 10 monclient: _renew_subs 2024-05-04T17:45:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.407+0000 7f72d7045640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.407+0000 7f72d7045640 1 -- 172.21.15.19:0/846475836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d013b4c0 con 0x7f72d0146050 2024-05-04T17:45:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.407+0000 7f72d7045640 1 librados: init done 2024-05-04T17:45:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.407+0000 7f72d7045640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.407+0000 7f72d7045640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.407+0000 7f72d7045640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:30.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.407+0000 7f72d7045640 1 -- 172.21.15.19:0/846475836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d0068b10 con 0x7f72d0146050 2024-05-04T17:45:30.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.659+0000 7f72cd7fa640 1 -- 172.21.15.19:0/846475836 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f72c801a070 con 0x7f72d0146050 2024-05-04T17:45:30.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.660+0000 7f72cd7fa640 1 --2- 172.21.15.19:0/846475836 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f72b406d810 0x7f72b406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:45:30.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.660+0000 7f72d4de1640 1 --2- 172.21.15.19:0/846475836 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f72b406d810 0x7f72b406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:45:30.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.660+0000 7f72d4de1640 10 monclient: get_auth_request con 0x7f72b406d810 auth_method 0 2024-05-04T17:45:30.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.660+0000 7f72cd7fa640 10 client.4438.objecter ms_handle_connect 0x7f72b406d810 2024-05-04T17:45:30.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.660+0000 7f72d4de1640 1 --2- 172.21.15.19:0/846475836 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f72b406d810 0x7f72b406fcd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f72c4036000 tx=0x7f72c403d040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:45:30.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.664+0000 7f72cd7fa640 1 -- 172.21.15.19:0/846475836 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f72c8057300 con 0x7f72d0146050 2024-05-04T17:45:30.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.664+0000 7f72cd7fa640 10 client.4438.objecter ms_dispatch 0x7f72d0000b90 osd_map(25..25 src has 1..25) 2024-05-04T17:45:30.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.664+0000 7f72cd7fa640 3 client.4438.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:45:30.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.664+0000 7f72cd7fa640 3 client.4438.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:45:30.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.664+0000 7f72cd7fa640 20 client.4438.objecter dump_active .. 0 homeless 2024-05-04T17:45:30.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.687+0000 7f72cd7fa640 1 -- 172.21.15.19:0/846475836 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f72c8059260 con 0x7f72d0146050 2024-05-04T17:45:30.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.687+0000 7f72cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T17:45:30.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.687+0000 7f72cd7fa640 10 monclient: _finish_command 1 = system:0 2024-05-04T17:45:30.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.927+0000 7f72d7045640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-05-04T17:45:30.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.927+0000 7f72d7045640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-05-04T17:45:30.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.927+0000 7f72d7045640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T17:45:30.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.927+0000 7f72d7045640 1 -- 172.21.15.19:0/846475836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f72d00b8110 con 0x7f72d0146050 2024-05-04T17:45:30.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.932+0000 7f72cd7fa640 1 -- 172.21.15.19:0/846475836 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f72c805d560 con 0x7f72d0146050 2024-05-04T17:45:30.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.932+0000 7f72cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-05-04T17:45:30.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.932+0000 7f72cd7fa640 10 monclient: _finish_command 2 = system:0 2024-05-04T17:45:30.933 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T17:45:30.933 INFO:teuthology.orchestra.run.smithi019.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-04T17:45:30.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 10 librados: watch_flush enter 2024-05-04T17:45:30.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 10 librados: watch_flush exit 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 20 client.4438.objecter shutdown clearing up homeless session... 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 10 client.4438.objecter successfully canceled tick 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 1 -- 172.21.15.19:0/846475836 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f72b406d810 msgr2=0x7f72b406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 1 --2- 172.21.15.19:0/846475836 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f72b406d810 0x7f72b406fcd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f72c4036000 tx=0x7f72c403d040 comp rx=0 tx=0).stop 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 10 monclient: shutdown 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 1 -- 172.21.15.19:0/846475836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72d0146050 msgr2=0x7f72d00b7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 1 --2- 172.21.15.19:0/846475836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72d0146050 0x7f72d00b7680 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f72c800c9f0 tx=0x7f72c800ceb0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 1 -- 172.21.15.19:0/846475836 shutdown_connections 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 1 --2- 172.21.15.19:0/846475836 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f72b406d810 0x7f72b406fcd0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 1 --2- 172.21.15.19:0/846475836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72d0146050 0x7f72d00b7680 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 1 --2- 172.21.15.19:0/846475836 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f72d00b8f30 0x7f72d00ac340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 1 --2- 172.21.15.19:0/846475836 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f72d00aa8b0 0x7f72d00adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:45:30.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 1 -- 172.21.15.19:0/846475836 >> 172.21.15.19:0/846475836 conn(0x7f72d00676f0 msgr2=0x7f72d00b4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:45:30.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.934+0000 7f72d7045640 1 -- 172.21.15.19:0/846475836 shutdown_connections 2024-05-04T17:45:30.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.935+0000 7f72d7045640 1 -- 172.21.15.19:0/846475836 wait complete. 2024-05-04T17:45:30.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T17:45:30.935+0000 7f72d7045640 1 librados: shutdown 2024-05-04T17:45:30.946 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-04T17:45:30.946 INFO:teuthology.run_tasks:Running task workunit... 2024-05-04T17:45:30.960 INFO:tasks.workunit:Pulling workunits from ref f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:45:30.960 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-04T17:45:30.960 DEBUG:teuthology.orchestra.run.smithi019:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-04T17:45:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T17:45:30.975 INFO:teuthology.orchestra.run.smithi019.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-04T17:45:30.976 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-05-04T17:45:31.031 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-05-04T17:45:31.031 DEBUG:teuthology.orchestra.run.smithi019:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-05-04T17:45:31.088 INFO:tasks.workunit:timeout=6h 2024-05-04T17:45:31.088 INFO:tasks.workunit:cleanup=True 2024-05-04T17:45:31.088 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f273489c6dcd4bc88409993babd09dd99491162c 2024-05-04T17:45:31.143 INFO:tasks.workunit.client.0.smithi019.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-04T17:45:34.501 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:29.432 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:34.552 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:29.482 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:34.558 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:29.447 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:34.561 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:29.492 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:34.565 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:29.496 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:34.576 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:29.466 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:34.607 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:29.538 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:34.613 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:29.544 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:34.617 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:29.507 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:54.513 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:49.443 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:54.521 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:49.452 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:54.583 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:49.472 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:54.584 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:49.473 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:54.596 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:01:49.485 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:54.603 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:49.533 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:54.605 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:49.536 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:54.635 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:01:49.566 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:45:54.636 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:01:49.567 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:14.600 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:09.530 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:14.650 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:09.537 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:14.650 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:09.538 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:14.661 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:09.551 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:14.669 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:09.600 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:14.683 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:09.614 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:14.684 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:09.615 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:14.689 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:09.620 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:14.712 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:09.643 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:20.073 INFO:tasks.workunit.client.0.smithi019.stderr:Updating files: 94% (11399/12088) Updating files: 95% (11484/12088) Updating files: 96% (11605/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-05-04T17:46:20.529 INFO:tasks.workunit.client.0.smithi019.stderr:Note: switching to 'f273489c6dcd4bc88409993babd09dd99491162c'. 2024-05-04T17:46:20.530 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-05-04T17:46:20.530 INFO:tasks.workunit.client.0.smithi019.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-04T17:46:20.530 INFO:tasks.workunit.client.0.smithi019.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-04T17:46:20.530 INFO:tasks.workunit.client.0.smithi019.stderr:state without impacting any branches by switching back to a branch. 2024-05-04T17:46:20.530 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-05-04T17:46:20.531 INFO:tasks.workunit.client.0.smithi019.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-04T17:46:20.531 INFO:tasks.workunit.client.0.smithi019.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-04T17:46:20.531 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-05-04T17:46:20.531 INFO:tasks.workunit.client.0.smithi019.stderr: git switch -c 2024-05-04T17:46:20.531 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-05-04T17:46:20.531 INFO:tasks.workunit.client.0.smithi019.stderr:Or undo this operation with: 2024-05-04T17:46:20.532 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-05-04T17:46:20.532 INFO:tasks.workunit.client.0.smithi019.stderr: git switch - 2024-05-04T17:46:20.532 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-05-04T17:46:20.532 INFO:tasks.workunit.client.0.smithi019.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-04T17:46:20.532 INFO:tasks.workunit.client.0.smithi019.stderr: 2024-05-04T17:46:20.533 INFO:tasks.workunit.client.0.smithi019.stderr:HEAD is now at f273489c6dc Merge branch 'wip-65185-squid' of https://github.com/rzarzynski/ceph into wip-yuri11-testing-2024-05-02-1242-squid 2024-05-04T17:46:20.536 DEBUG:teuthology.orchestra.run.smithi019:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-04T17:46:20.598 INFO:tasks.workunit.client.0.smithi019.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-04T17:46:20.600 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-04T17:46:20.600 INFO:tasks.workunit.client.0.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-04T17:46:21.235 INFO:tasks.workunit.client.0.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-04T17:46:21.290 INFO:tasks.workunit.client.0.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-04T17:46:21.330 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-04T17:46:21.332 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-04T17:46:21.332 INFO:tasks.workunit.client.0.smithi019.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-04T17:46:21.371 INFO:tasks.workunit.client.0.smithi019.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-04T17:46:21.375 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-05-04T17:46:21.375 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-04T17:46:21.431 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-05-04T17:46:21.432 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-05-04T17:46:21.432 DEBUG:teuthology.orchestra.run.smithi019:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f273489c6dcd4bc88409993babd09dd99491162c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-05-04T17:46:21.490 INFO:tasks.workunit.client.0.smithi019.stderr:+ parallel=1 2024-05-04T17:46:21.490 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' '' = --serial ']' 2024-05-04T17:46:21.490 INFO:tasks.workunit.client.0.smithi019.stderr:+ crimson=0 2024-05-04T17:46:21.490 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' '' = --crimson ']' 2024-05-04T17:46:21.490 INFO:tasks.workunit.client.0.smithi019.stderr:+ color= 2024-05-04T17:46:21.490 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -t 1 ']' 2024-05-04T17:46:21.491 INFO:tasks.workunit.client.0.smithi019.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-05-04T17:46:21.491 INFO:tasks.workunit.client.0.smithi019.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-05-04T17:46:21.491 INFO:tasks.workunit.client.0.smithi019.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-05-04T17:46:21.492 INFO:tasks.workunit.client.0.smithi019.stderr:+ declare -A pids 2024-05-04T17:46:21.492 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.492 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.492 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_aio 2024-05-04T17:46:21.492 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_aio' 2024-05-04T17:46:21.492 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_aio 2024-05-04T17:46:21.492 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_aio 2024-05-04T17:46:21.493 INFO:tasks.workunit.client.0.smithi019.stdout:test api_aio on pid 38323 2024-05-04T17:46:21.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38323 2024-05-04T17:46:21.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_aio on pid 38323' 2024-05-04T17:46:21.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38323 2024-05-04T17:46:21.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_aio_pp 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_aio_pp' 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_aio_pp 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.497 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.498 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.498 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.498 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.498 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.498 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.498 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.498 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.498 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.498 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.498 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.498 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.499 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.499 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.499 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.499 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.499 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.499 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_aio_pp 2024-05-04T17:46:21.499 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38330 2024-05-04T17:46:21.499 INFO:tasks.workunit.client.0.smithi019.stdout:test api_aio_pp on pid 38330 2024-05-04T17:46:21.503 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_aio_pp on pid 38330' 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38330 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_io 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_io' 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_io 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.504 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.505 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.506 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_io 2024-05-04T17:46:21.507 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38339 2024-05-04T17:46:21.508 INFO:tasks.workunit.client.0.smithi019.stdout:test api_io on pid 38339 2024-05-04T17:46:21.512 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_io on pid 38339' 2024-05-04T17:46:21.512 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38339 2024-05-04T17:46:21.512 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.512 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_io_pp 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_io_pp' 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_io_pp 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.513 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.514 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.515 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.516 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_io_pp 2024-05-04T17:46:21.517 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38350 2024-05-04T17:46:21.517 INFO:tasks.workunit.client.0.smithi019.stdout:test api_io_pp on pid 38350 2024-05-04T17:46:21.522 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_io_pp on pid 38350' 2024-05-04T17:46:21.522 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38350 2024-05-04T17:46:21.522 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_asio 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_asio' 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_asio 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.523 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.524 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.525 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.525 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.525 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.525 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.525 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.525 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.525 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.525 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.525 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.525 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.525 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.526 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.526 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.526 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.526 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.526 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.526 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.526 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.526 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.526 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.526 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.526 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.527 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.528 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.529 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.529 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_asio 2024-05-04T17:46:21.529 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38365 2024-05-04T17:46:21.529 INFO:tasks.workunit.client.0.smithi019.stdout:test api_asio on pid 38365 2024-05-04T17:46:21.536 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_asio on pid 38365' 2024-05-04T17:46:21.536 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38365 2024-05-04T17:46:21.536 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.536 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.536 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-05-04T17:46:21.536 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_list 2024-05-04T17:46:21.536 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.536 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.536 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.536 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.536 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.536 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_list' 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_list 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.537 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.538 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.539 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.540 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.541 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.542 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.542 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.542 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.542 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.542 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.542 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.542 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.542 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.542 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.542 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.542 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.542 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.543 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.543 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.543 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.543 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.543 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.543 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.543 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.543 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.543 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_list 2024-05-04T17:46:21.543 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38323/exe 2024-05-04T17:46:21.543 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38383 2024-05-04T17:46:21.544 INFO:tasks.workunit.client.0.smithi019.stdout:test api_list on pid 38383 2024-05-04T17:46:21.557 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_list on pid 38383' 2024-05-04T17:46:21.557 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38383 2024-05-04T17:46:21.557 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.558 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.558 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-05-04T17:46:21.558 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_lock 2024-05-04T17:46:21.558 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.558 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_lock' 2024-05-04T17:46:21.558 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.558 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.558 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.559 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.560 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.561 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.562 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_lock 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.563 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_aio.log 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.564 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.565 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.566 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_aio: /' 2024-05-04T17:46:21.567 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.567 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.567 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38330/exe 2024-05-04T17:46:21.567 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_lock 2024-05-04T17:46:21.567 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.567 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.567 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.567 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38405 2024-05-04T17:46:21.567 INFO:tasks.workunit.client.0.smithi019.stdout:test api_lock on pid 38405 2024-05-04T17:46:21.574 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_lock on pid 38405' 2024-05-04T17:46:21.574 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.574 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38405 2024-05-04T17:46:21.574 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.575 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_lock_pp 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_lock_pp' 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-05-04T17:46:21.576 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.577 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.577 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.577 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.577 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.577 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_lock_pp 2024-05-04T17:46:21.577 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.577 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.577 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.577 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.577 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.577 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.577 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.578 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.579 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.580 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.580 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.580 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.580 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.580 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.580 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.580 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.580 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.580 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.580 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.580 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.580 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_aio_pp: /' 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.581 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.582 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38339/exe 2024-05-04T17:46:21.582 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.582 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.582 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.582 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_lock_pp 2024-05-04T17:46:21.582 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.582 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.582 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.582 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38423 2024-05-04T17:46:21.582 INFO:tasks.workunit.client.0.smithi019.stdout:test api_lock_pp on pid 38423 2024-05-04T17:46:21.590 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_lock_pp on pid 38423' 2024-05-04T17:46:21.590 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38423 2024-05-04T17:46:21.591 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.591 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.591 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.591 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.591 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.591 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-05-04T17:46:21.591 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.591 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.591 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.592 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.592 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.592 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.592 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_misc 2024-05-04T17:46:21.592 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.592 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.593 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.593 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.593 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.593 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.593 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.593 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.593 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.594 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.594 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.594 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_misc' 2024-05-04T17:46:21.594 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.594 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.594 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.594 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.594 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.595 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.595 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.595 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.595 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.595 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.595 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.595 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.595 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.595 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.595 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.595 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.596 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.596 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.596 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.596 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.596 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.596 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.596 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.596 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_misc 2024-05-04T17:46:21.596 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.597 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.598 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.598 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.598 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.598 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.598 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.598 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.598 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.598 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.598 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.598 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.598 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.598 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.599 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.599 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.599 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.599 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.599 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.599 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.599 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.599 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.599 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.599 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.599 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.599 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.600 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.600 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.600 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_io: /' 2024-05-04T17:46:21.600 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.600 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.600 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.600 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.600 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38350/exe 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_misc 2024-05-04T17:46:21.601 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38441 2024-05-04T17:46:21.602 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_misc on pid 38441' 2024-05-04T17:46:21.602 INFO:tasks.workunit.client.0.smithi019.stdout:test api_misc on pid 38441 2024-05-04T17:46:21.622 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38441 2024-05-04T17:46:21.622 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.622 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.622 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.622 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.622 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.622 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.623 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_io.log 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.624 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.625 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.625 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.625 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.625 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.625 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.625 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.625 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.625 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.625 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.625 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.625 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.625 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.626 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.626 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.626 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.626 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.626 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.626 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.626 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.626 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.626 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.626 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.626 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.626 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.627 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.627 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.627 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.627 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.627 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.627 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.627 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.627 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.627 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.627 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.627 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.628 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.629 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.630 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.630 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_io_pp: /' 2024-05-04T17:46:21.630 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.630 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.630 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.630 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.630 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.631 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.632 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.632 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.632 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-05-04T17:46:21.632 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38365/exe 2024-05-04T17:46:21.632 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.632 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-05-04T17:46:21.632 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.632 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.632 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.632 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.632 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_misc_pp 2024-05-04T17:46:21.633 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.633 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.633 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.633 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.633 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.633 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.633 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_misc_pp' 2024-05-04T17:46:21.633 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.633 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.634 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.634 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.634 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.634 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.634 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.634 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.634 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.634 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.634 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.635 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.636 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.636 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.636 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.636 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.636 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38383/exe 2024-05-04T17:46:21.636 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.637 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.637 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.637 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_asio: /' 2024-05-04T17:46:21.637 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-05-04T17:46:21.637 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.637 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.638 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.638 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.638 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.638 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.638 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.638 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.638 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.638 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.638 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.638 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.638 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.639 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_asio.log 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_list: /' 2024-05-04T17:46:21.640 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_list.log 2024-05-04T17:46:21.641 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.641 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.641 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.641 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.641 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.641 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.642 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.642 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.642 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.642 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.642 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.642 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.643 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.643 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-05-04T17:46:21.643 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.643 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.643 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.643 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_misc_pp 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.644 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.645 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.645 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.645 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.645 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.645 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.645 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.645 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.645 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.646 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.646 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.646 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.646 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.646 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.646 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.646 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.646 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.646 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.646 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_misc_pp 2024-05-04T17:46:21.647 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38482 2024-05-04T17:46:21.647 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_misc_pp on pid 38482' 2024-05-04T17:46:21.647 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.647 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.647 INFO:tasks.workunit.client.0.smithi019.stdout:test api_misc_pp on pid 38482 2024-05-04T17:46:21.652 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.652 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38482 2024-05-04T17:46:21.652 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.652 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.652 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_tier_pp 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_tier_pp' 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.653 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.654 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.654 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.654 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.654 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.654 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.654 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.654 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.654 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.655 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.655 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.655 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.655 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.655 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.655 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.655 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.655 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.655 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.655 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.656 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.657 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.657 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_tier_pp 2024-05-04T17:46:21.657 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.657 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.657 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.657 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.657 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.657 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.657 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_tier_pp 2024-05-04T17:46:21.657 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38492 2024-05-04T17:46:21.657 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_tier_pp on pid 38492' 2024-05-04T17:46:21.657 INFO:tasks.workunit.client.0.smithi019.stdout:test api_tier_pp on pid 38492 2024-05-04T17:46:21.661 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38492 2024-05-04T17:46:21.661 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38423/exe 2024-05-04T17:46:21.661 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.661 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.661 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.661 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_pool 2024-05-04T17:46:21.661 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.661 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_pool' 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_pool 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.662 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.663 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.663 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.663 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.663 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.663 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.663 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_pool 2024-05-04T17:46:21.663 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.663 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.663 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.663 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.663 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.663 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38501 2024-05-04T17:46:21.664 INFO:tasks.workunit.client.0.smithi019.stdout:test api_pool on pid 38501 2024-05-04T17:46:21.680 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_pool on pid 38501' 2024-05-04T17:46:21.680 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.681 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38501 2024-05-04T17:46:21.681 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.681 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.681 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.681 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.681 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.681 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.681 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-05-04T17:46:21.682 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.682 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.682 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.682 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.682 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.682 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.682 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.682 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.683 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.683 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.683 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.683 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.683 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.683 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.683 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.683 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.683 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.683 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.684 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.684 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.684 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.684 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.684 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.684 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.684 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.684 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.685 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.685 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.685 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.685 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.685 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.685 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.685 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.685 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-05-04T17:46:21.685 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.686 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.686 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.686 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.686 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.686 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.686 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.686 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.686 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.686 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.687 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.687 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.687 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.687 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.687 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.687 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.687 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.687 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.688 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.688 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.688 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.688 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_lock_pp: /' 2024-05-04T17:46:21.688 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.688 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.688 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.688 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-05-04T17:46:21.689 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.689 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.689 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.689 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.689 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.689 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.689 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.689 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.689 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.690 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.690 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.690 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.690 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.690 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.690 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.690 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.690 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.691 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.691 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.691 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.691 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.691 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_snapshots 2024-05-04T17:46:21.691 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.691 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.691 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.691 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.692 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.692 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.692 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.692 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.692 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.692 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.692 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.692 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.692 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.693 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.693 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.693 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.693 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.693 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.693 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.693 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.693 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.693 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.694 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.694 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.694 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.694 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.694 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.694 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.694 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.694 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.694 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.694 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.695 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.696 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.696 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.696 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.696 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.696 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.696 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.696 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.696 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_snapshots' 2024-05-04T17:46:21.696 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_snapshots 2024-05-04T17:46:21.696 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.696 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.696 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.697 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.697 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.697 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.697 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.697 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.697 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.697 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.697 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_snapshots 2024-05-04T17:46:21.697 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.697 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38558 2024-05-04T17:46:21.697 INFO:tasks.workunit.client.0.smithi019.stdout:test api_snapshots on pid 38558 2024-05-04T17:46:21.703 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_snapshots on pid 38558' 2024-05-04T17:46:21.703 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38558 2024-05-04T17:46:21.703 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.703 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.703 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38405/exe 2024-05-04T17:46:21.703 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_snapshots_pp 2024-05-04T17:46:21.703 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_snapshots_pp' 2024-05-04T17:46:21.703 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_snapshots_pp 2024-05-04T17:46:21.703 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.703 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.703 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.703 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.704 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.705 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_lock.log 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.706 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.707 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.707 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.707 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_snapshots_pp 2024-05-04T17:46:21.707 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.707 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.707 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.707 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38574 2024-05-04T17:46:21.707 INFO:tasks.workunit.client.0.smithi019.stdout:test api_snapshots_pp on pid 38574 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_snapshots_pp on pid 38574' 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38574 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_stat 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_stat' 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_lock: /' 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_stat 2024-05-04T17:46:21.709 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_stat 2024-05-04T17:46:21.710 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.710 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.710 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.710 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38582 2024-05-04T17:46:21.710 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_stat on pid 38582' 2024-05-04T17:46:21.710 INFO:tasks.workunit.client.0.smithi019.stdout:test api_stat on pid 38582 2024-05-04T17:46:21.720 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38582 2024-05-04T17:46:21.720 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.720 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.720 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38441/exe 2024-05-04T17:46:21.720 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.720 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.720 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.720 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-05-04T17:46:21.720 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_stat_pp 2024-05-04T17:46:21.720 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_stat_pp' 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_stat_pp 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.721 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.722 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.723 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.724 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.725 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.726 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.727 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.727 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.727 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.727 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.727 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.727 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.727 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.727 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.727 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.727 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.727 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.727 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.728 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.728 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.728 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.728 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.728 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.728 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.728 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_stat_pp 2024-05-04T17:46:21.728 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38501/exe 2024-05-04T17:46:21.728 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38605 2024-05-04T17:46:21.728 INFO:tasks.workunit.client.0.smithi019.stdout:test api_stat_pp on pid 38605 2024-05-04T17:46:21.740 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_stat_pp on pid 38605' 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38605 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.741 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.742 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.743 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_misc: /' 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.744 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_watch_notify 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_misc.log 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_watch_notify' 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.745 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.746 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.747 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.748 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.749 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_watch_notify 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.750 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.751 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.751 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.751 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.751 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.751 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.751 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.751 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.751 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.751 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.751 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.751 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [----------] Global test environment set-up. 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [----------] 12 tests from AsioRados 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.603+0000 7f3e39d86780 10 monclient: get_monmap_and_config 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.603+0000 7f3e39d86780 10 monclient: build_initial_monmap 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.603+0000 7f3e39d86780 10 monclient: monmap: 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: epoch 0 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: last_changed 0.000000 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: created 0.000000 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: min_mon_release 0 (unknown) 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: election_strategy: 1 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:46:21.752 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.603+0000 7f3e39d86780 1 Processor -- start 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.603+0000 7f3e39d86780 1 -- start start 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.603+0000 7f3e39d86780 10 monclient: init 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 10 monclient: _reopen_session rank -1 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 10 monclient: _add_conns ranks=[2,0,1] 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x55dfbde37740 0x55dfbdf013d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 10 monclient(hunting): picked mon.noname-c con 0x55dfbde37740 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55dfbde3b590 0x55dfbde4fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 10 monclient(hunting): picked mon.noname-a con 0x55dfbde3b590 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfbde50b70 0x55dfbde50080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 10 monclient(hunting): picked mon.noname-b con 0x55dfbde50b70 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 10 monclient(hunting): start opening mon connection 2024-05-04T17:46:21.753 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55dfbdde8c70 con 0x55dfbde3b590 2024-05-04T17:46:21.754 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 10 monclient(hunting): start opening mon connection 2024-05-04T17:46:21.754 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55dfbdd92600 con 0x55dfbde50b70 2024-05-04T17:46:21.754 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 10 monclient(hunting): start opening mon connection 2024-05-04T17:46:21.754 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x55dfbde37b70 con 0x55dfbde37740 2024-05-04T17:46:21.754 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 10 monclient(hunting): _renew_subs 2024-05-04T17:46:21.754 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.604+0000 7f3e39d86780 10 monclient(hunting): authenticate will time out at 1403.719160s 2024-05-04T17:46:21.762 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_pool.log 2024-05-04T17:46:21.762 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_pool: /' 2024-05-04T17:46:21.762 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.762 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.762 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.762 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.762 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.762 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.762 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.762 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.763 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.764 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.765 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38482/exe 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.766 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.767 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_watch_notify 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.768 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38679 2024-05-04T17:46:21.769 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-0test api_watch_notify on pid 38679 2024-05-04T17:46:21.774 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_watch_notify on pid 38679' 2024-05-04T17:46:21.774 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38679 2024-05-04T17:46:21.774 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.774 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.774 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.774 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.774 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.774 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.774 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.774 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.775 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.776 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_watch_notify_pp 2024-05-04T17:46:21.776 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_watch_notify_pp' 2024-05-04T17:46:21.776 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38492/exe 2024-05-04T17:46:21.776 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.776 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.776 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.776 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.776 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.776 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.776 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.776 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.776 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.777 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.777 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.777 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.777 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.777 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.777 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.777 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.777 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.777 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.777 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.778 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.778 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.778 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.778 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.778 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_watch_notify_pp 2024-05-04T17:46:21.778 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-05-04T17:46:21.778 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-05-04T17:46:21.778 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_misc_pp: /' 2024-05-04T17:46:21.778 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.778 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.778 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.779 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.779 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.779 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.779 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.779 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.779 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.779 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.779 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-05-04T17:46:21.780 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_watch_notify_pp 2024-05-04T17:46:21.780 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-05-04T17:46:21.780 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38696 2024-05-04T17:46:21.780 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_watch_notify_pp on pid 38696' 2024-05-04T17:46:21.781 INFO:tasks.workunit.client.0.smithi019.stdout:test api_watch_notify_pp on pid 38696 2024-05-04T17:46:21.785 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38696 2024-05-04T17:46:21.785 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.785 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.785 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.785 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.785 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.785 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.785 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.785 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_cmd 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.786 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.787 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.787 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_cmd' 2024-05-04T17:46:21.787 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.787 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.787 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.787 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.787 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.787 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-05-04T17:46:21.787 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.787 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.787 INFO:tasks.workunit.client.0.smithi019.stdout:4T17:46:21.606+0000 7f3e3808a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfbde50b70 0x55dfbde50080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:46:21.787 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.610+0000 7f3e3808a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfbde50b70 0x55dfbde50080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:35100/0 (socket says 172.21.15.19:35100) 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.610+0000 7f3e3808a640 1 -- 172.21.15.19:0/3399638779 learned_addr learned my addr 172.21.15.19:0/3399638779 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.610+0000 7f3e3808a640 10 monclient(hunting): get_auth_request con 0x55dfbde50b70 auth_method 0 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.610+0000 7f3e3808a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.610+0000 7f3e3808a640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.610+0000 7f3e3808a640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.611+0000 7f3e3808a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.611+0000 7f3e3808a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.611+0000 7f3e3808a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.613+0000 7f3e3808a640 10 monclient(hunting): handle_auth_done global_id 4459 payload 274 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.613+0000 7f3e3808a640 1 -- 172.21.15.19:0/3399638779 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x55dfbde37740 msgr2=0x55dfbdf013d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.613+0000 7f3e3808a640 1 --2- 172.21.15.19:0/3399638779 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x55dfbde37740 0x55dfbdf013d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.613+0000 7f3e3808a640 1 -- 172.21.15.19:0/3399638779 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55dfbde3b590 msgr2=0x55dfbde4fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.613+0000 7f3e3808a640 1 --2- 172.21.15.19:0/3399638779 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55dfbde3b590 0x55dfbde4fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.613+0000 7f3e3808a640 10 monclient: _finish_hunting 0 2024-05-04T17:46:21.788 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.613+0000 7f3e3808a640 1 monclient: found mon.noname-b 2024-05-04T17:46:21.789 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.613+0000 7f3e3808a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:46:21.789 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.613+0000 7f3e3808a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T17:46:21.789 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.613+0000 7f3e3808a640 1 -- 172.21.15.19:0/3399638779 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfbdf00a80 con 0x55dfbde50b70 2024-05-04T17:46:21.789 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.615+0000 7f3e3808a640 1 --2- 172.21.15.19:0/3399638779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfbde50b70 0x55dfbde50080 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c001db0 tx=0x7f3e2c00b6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61fe8669125c421d server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:46:21.791 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.791 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.791 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.792 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.793 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.793 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.793 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.793 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.793 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.793 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.793 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.793 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.793 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.793 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.793 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 1 -- 172.21.15.19:0/3399638779 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c012020 con 0x55dfbde50b70 2024-05-04T17:46:21.793 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 10 monclient: dump: 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: epoch 1 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: min_mon_release 19 (squid) 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: election_strategy: 1 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 10 monclient: _finish_auth 0 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 10 monclient: _check_auth_tickets 2024-05-04T17:46:21.794 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 1 -- 172.21.15.19:0/3399638779 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e2c00e670 con 0x55dfbde50b70 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 10 monclient: handle_config config(0 keys) 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 1 -- 172.21.15.19:0/3399638779 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c00eaa0 con 0x55dfbde50b70 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.630+0000 7f3e36086640 10 monclient: dump: 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: epoch 1 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: min_mon_release 19 (squid) 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: election_strategy: 1 2024-05-04T17:46:21.795 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:46:21.796 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:46:21.796 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:46:21.796 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.796 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.631+0000 7f3e39d86780 5 monclient: authenticate success, global_id 4459 2024-05-04T17:46:21.796 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.631+0000 7f3e39d86780 10 monclient: get_monmap_and_config success 2024-05-04T17:46:21.796 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.631+0000 7f3e39d86780 4 set_mon_vals no callback set 2024-05-04T17:46:21.796 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.632+0000 7f3e39d86780 10 monclient: shutdown 2024-05-04T17:46:21.796 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.632+0000 7f3e39d86780 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:46:21.796 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.632+0000 7f3e39d86780 1 -- 172.21.15.19:0/3399638779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfbde50b70 msgr2=0x55dfbde50080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:46:21.796 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.632+0000 7f3e39d86780 1 --2- 172.21.15.19:0/3399638779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfbde50b70 0x55dfbde50080 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c001db0 tx=0x7f3e2c00b6c0 comp rx=0 tx=0).stop 2024-05-04T17:46:21.799 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.799 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.799 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.799 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.799 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.799 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.799 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_tier_pp: /' 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.800 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.801 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.801 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.801 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.801 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.801 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.801 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.801 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.634+0000 7f3e39d86780 1 -- 172.21.15.19:0/3399638779 shutdown_connections 2024-05-04T17:46:21.801 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.634+0000 7f3e39d86780 1 --2- 172.21.15.19:0/3399638779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfbde50b70 0x55dfbde50080 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:46:21.801 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.634+0000 7f3e39d86780 1 --2- 172.21.15.19:0/3399638779 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55dfbde3b590 0x55dfbde4fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:46:21.801 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.634+0000 7f3e39d86780 1 --2- 172.21.15.19:0/3399638779 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x55dfbde37740 0x55dfbdf013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:46:21.801 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.634+0000 7f3e39d86780 1 -- 172.21.15.19:0/3399638779 >> 172.21.15.19:0/3399638779 conn(0x55dfbde797f0 msgr2=0x55dfbdf02070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:46:21.802 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.634+0000 7f3e39d86780 1 -- 172.21.15.19:0/3399638779 shutdown_connections 2024-05-04T17:46:21.802 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.634+0000 7f3e39d86780 1 -- 172.21.15.19:0/3399638779 wait complete. 2024-05-04T17:46:21.802 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.634+0000 7f3e39d86780 10 monclient: build_initial_monmap 2024-05-04T17:46:21.802 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.634+0000 7f3e39d86780 10 monclient: monmap: 2024-05-04T17:46:21.802 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: epoch 0 2024-05-04T17:46:21.802 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-05-04T17:46:21.802 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: last_changed 0.000000 2024-05-04T17:46:21.803 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: created 0.000000 2024-05-04T17:46:21.803 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: min_mon_release 0 (unknown) 2024-05-04T17:46:21.803 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: election_strategy: 1 2024-05-04T17:46:21.803 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T17:46:21.803 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T17:46:21.803 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T17:46:21.803 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.803 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.634+0000 7f3e39d86780 1 librados: starting msgr at 2024-05-04T17:46:21.803 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.634+0000 7f3e39d86780 1 librados: starting objecter 2024-05-04T17:46:21.803 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.636+0000 7f3e39d86780 1 Processor -- start 2024-05-04T17:46:21.803 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.636+0000 7f3e39d86780 1 -- start start 2024-05-04T17:46:21.803 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.636+0000 7f3e39d86780 1 librados: setting wanted keys 2024-05-04T17:46:21.804 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.636+0000 7f3e39d86780 1 librados: calling monclient init 2024-05-04T17:46:21.804 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.636+0000 7f3e39d86780 10 monclient: init 2024-05-04T17:46:21.804 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 10 monclient: _reopen_session rank -1 2024-05-04T17:46:21.804 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T17:46:21.804 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55dfbde37740 0x55dfbdfb06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:46:21.804 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 10 monclient(hunting): picked mon.noname-a con 0x55dfbde37740 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T17:46:21.804 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x55dfbde3b590 0x55dfbdfae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:46:21.804 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 10 monclient(hunting): picked mon.noname-c con 0x55dfbde3b590 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T17:46:21.805 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.805 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.805 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.805 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_cmd 2024-05-04T17:46:21.805 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.806 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.807 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_cmd 2024-05-04T17:46:21.810 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38739 2024-05-04T17:46:21.810 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_cmd on pid 38739' 2024-05-04T17:46:21.810 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfbde50b70 0x55dfbdeede00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0test api_cmd on pid 38739 2024-05-04T17:46:21.823 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38739 2024-05-04T17:46:21.823 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.823 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.823 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.823 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.823 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.823 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.823 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.823 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.823 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.823 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-05-04T17:46:21.823 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.824 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.825 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.825 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.825 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.825 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_cmd_pp 2024-05-04T17:46:21.825 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_cmd_pp' 2024-05-04T17:46:21.825 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.825 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.825 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.825 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.825 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.825 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.825 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.826 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.827 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.828 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.828 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.828 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.828 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.828 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.828 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.828 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_cmd_pp 2024-05-04T17:46:21.828 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.829 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.830 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.831 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.832 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.833 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38574/exe 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.834 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.835 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.835 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.835 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.835 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_cmd_pp 2024-05-04T17:46:21.835 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.835 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.835 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38768 2024-05-04T17:46:21.835 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_cmd_pp on pid 38768' 2024-05-04T17:46:21.835 INFO:tasks.workunit.client.0.smithi019.stdout:test api_cmd_pp on pid 38768 2024-05-04T17:46:21.853 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.853 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38768 2024-05-04T17:46:21.853 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.853 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.853 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.853 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.853 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.853 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.853 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.854 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.855 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.856 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_service 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.857 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.858 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38582/exe 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_service' 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.859 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.860 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.861 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_stat: /' 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38558/exe 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.862 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.863 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.864 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38696/exe 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.865 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.866 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_service 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.867 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.868 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.869 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.869 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.869 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.869 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.869 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-05-04T17:46:21.869 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_stat.log 2024-05-04T17:46:21.869 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-05-04T17:46:21.869 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_snapshots: /' 2024-05-04T17:46:21.869 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.869 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.869 INFO:tasks.workunit.client.0.smithi019.stdout: crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 10 monclient(hunting): picked mon.noname-b con 0x55dfbde50b70 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 10 monclient(hunting): start opening mon connection 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x55dfbdfae4c0 con 0x55dfbde37740 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 10 monclient(hunting): start opening mon connection 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55dfbdf037d0 con 0x55dfbde50b70 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 10 monclient(hunting): start opening mon connection 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x55dfbdf02f70 con 0x55dfbde3b590 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 10 monclient(hunting): _renew_subs 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.637+0000 7f3e39d86780 10 monclient(hunting): authenticate will time out at 1403.752081s 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.674+0000 7f3e37889640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55dfbde37740 0x55dfbdfb06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.677+0000 7f3e37889640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55dfbde37740 0x55dfbdfb06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:45516/0 (socket says 172.21.15.19:45516) 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.677+0000 7f3e37889640 1 -- 172.21.15.19:0/1329173258 learned_addr learned my addr 172.21.15.19:0/1329173258 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.677+0000 7f3e37889640 10 monclient(hunting): get_auth_request con 0x55dfbde37740 auth_method 0 2024-05-04T17:46:21.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.677+0000 7f3e37889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:46:21.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.677+0000 7f3e37889640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:46:21.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.677+0000 7f3e37889640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:46:21.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.679+0000 7f3e37889640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T17:46:21.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.679+0000 7f3e37889640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:46:21.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.679+0000 7f3e37889640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:46:21.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.684+0000 7f3e37088640 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x55dfbde3b590 0x55dfbdfae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:46:21.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.684+0000 7f3e37088640 10 monclient(hunting): get_auth_request con 0x55dfbde3b590 auth_method 0 2024-05-04T17:46:21.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.684+0000 7f3e37088640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T17:46:21.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.684+0000 7f3e37088640 10 monclient(hunting): _init_auth method 2 2024-05-04T17:46:21.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.684+0000 7f3e37088640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T17:46:21.876 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.876 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.876 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.876 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.876 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.876 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.876 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.876 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.876 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.876 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.876 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.877 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.878 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_service 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.879 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38836 2024-05-04T17:46:21.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.685+0000 7f3e37088640 10 monclient(hunting): hatest api_service on pid 38836 2024-05-04T17:46:21.894 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_service on pid 38836' 2024-05-04T17:46:21.894 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38836 2024-05-04T17:46:21.894 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.894 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.894 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.894 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.894 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.894 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.894 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.894 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.895 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38679/exe 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.896 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.897 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_watch_notify: /' 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.898 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_service_pp 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_service_pp' 2024-05-04T17:46:21.899 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38605/exe 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.900 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_service_pp 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.901 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.902 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.903 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.903 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.903 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.903 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.903 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.903 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.903 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.903 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.903 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.903 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.903 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38739/exe 2024-05-04T17:46:21.904 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.905 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_service_pp 2024-05-04T17:46:21.905 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38884 2024-05-04T17:46:21.905 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.905 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.905 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.905 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.905 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.905 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.905 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-05-04T17:46:21.905 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_service_pp on pid 38884' 2024-05-04T17:46:21.905 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.905 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.906 INFO:tasks.workunit.client.0.smithi019.stdout:test api_service_pp on pid 38884 2024-05-04T17:46:21.914 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38884 2024-05-04T17:46:21.914 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.914 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.914 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.915 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_stat_pp: /' 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_c_write_operations 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_c_write_operations' 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.916 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.917 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_cmd.log 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.918 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.919 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.919 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.919 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.919 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_cmd: /' 2024-05-04T17:46:21.919 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.919 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.922 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.922 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.922 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.922 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.922 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.922 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.922 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.922 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_c_write_operations 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.923 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.924 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.924 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.924 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.924 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.924 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.924 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.924 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_c_write_operations 2024-05-04T17:46:21.924 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38946 2024-05-04T17:46:21.924 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_c_write_operations on pid 38946' 2024-05-04T17:46:21.924 INFO:tasks.workunit.client.0.smithi019.stdout:test api_c_write_operations on pid 38946 2024-05-04T17:46:21.932 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38946 2024-05-04T17:46:21.932 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.932 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.932 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_c_read_operations 2024-05-04T17:46:21.932 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_c_read_operations' 2024-05-04T17:46:21.932 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_c_read_operations 2024-05-04T17:46:21.932 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.932 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.932 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38768/exe 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.933 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.934 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.935 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.936 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.937 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.938 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.938 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.938 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.938 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.938 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.938 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_c_read_operations 2024-05-04T17:46:21.938 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.938 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.938 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38971 2024-05-04T17:46:21.938 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_c_read_operations on pid 38971' 2024-05-04T17:46:21.938 INFO:tasks.workunit.client.0.smithi019.stdout:test api_c_read_operations on pid 38971 2024-05-04T17:46:21.947 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=38971 2024-05-04T17:46:21.947 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.948 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38884/exe 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-05-04T17:46:21.949 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_service_pp: /' 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.950 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.951 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s api_cls_remote_reads 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' api_cls_remote_reads' 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo api_cls_remote_reads 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.952 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=api_cls_remote_reads 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:21.953 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39006 2024-05-04T17:46:21.954 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:21.954 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test api_cls_remote_reads on pid 39006' 2024-05-04T17:46:21.954 INFO:tasks.workunit.client.0.smithi019.stdout:test api_cls_remote_reads on pid 39006 2024-05-04T17:46:21.961 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39006 2024-05-04T17:46:21.961 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.961 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.961 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.961 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.961 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.961 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.961 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.961 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.962 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.963 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38946/exe 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:21.964 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.965 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.966 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s list_parallel 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' list_parallel' 2024-05-04T17:46:21.967 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.968 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.968 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.968 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-05-04T17:46:21.968 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-05-04T17:46:21.968 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.968 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-05-04T17:46:21.968 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo list_parallel 2024-05-04T17:46:21.968 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=list_parallel 2024-05-04T17:46:21.968 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39030 2024-05-04T17:46:21.968 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test list_parallel on pid 39030' 2024-05-04T17:46:21.968 INFO:tasks.workunit.client.0.smithi019.stdout:test list_parallel on pid 39030 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39030 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38836/exe 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.977 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.978 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.979 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:21.980 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_service.log 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s open_pools_parallel 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' open_pools_parallel' 2024-05-04T17:46:21.981 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo open_pools_parallel 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_service: /' 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.982 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:21.983 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=open_pools_parallel 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39060 2024-05-04T17:46:21.984 INFO:tasks.workunit.client.0.smithi019.stdout:test open_pools_parallel on pid 39060 2024-05-04T17:46:21.995 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test open_pools_parallel on pid 39060' 2024-05-04T17:46:21.995 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39060 2024-05-04T17:46:21.995 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-05-04T17:46:21.995 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:21.995 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:21.995 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.995 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s delete_pools_parallel 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' delete_pools_parallel' 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:21.996 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:21.997 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:21.998 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:21.999 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.000 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.000 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.001 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.002 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.003 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39006/exe 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.004 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.005 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.005 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.005 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.005 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.005 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.005 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo delete_pools_parallel 2024-05-04T17:46:22.005 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.005 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=delete_pools_parallel 2024-05-04T17:46:22.005 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39088 2024-05-04T17:46:22.005 INFO:tasks.workunit.client.0.smithi019.stdout:test delete_pools_parallel on pid 39088 2024-05-04T17:46:22.024 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test delete_pools_parallel on pid 39088' 2024-05-04T17:46:22.024 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39088 2024-05-04T17:46:22.024 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.024 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.024 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.024 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.024 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.024 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.024 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.024 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.024 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.025 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.026 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.027 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.028 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.029 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/38971/exe 2024-05-04T17:46:22.030 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s cls 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.031 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' cls' 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.032 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.033 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.034 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.035 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo cls 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39030/exe 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.036 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.037 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.038 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.039 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.039 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=cls 2024-05-04T17:46:22.039 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39145 2024-05-04T17:46:22.039 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test cls on pid 39145' 2024-05-04T17:46:22.039 INFO:tasks.workunit.client.0.smithi019.stdout:test cls on pid 39145 2024-05-04T17:46:22.042 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39145 2024-05-04T17:46:22.042 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.042 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-05-04T17:46:22.042 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.042 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s cmd 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' cmd' 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39060/exe 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo cmd 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_list_parallel.log 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ list_parallel: /' 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=cmd 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.043 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.044 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.044 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.044 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.044 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.044 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.044 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.044 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39155 2024-05-04T17:46:22.044 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.044 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.044 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.044 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.044 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test cmd on pid 39155' 2024-05-04T17:46:22.045 INFO:tasks.workunit.client.0.smithi019.stdout:test cmd on pid 39155 2024-05-04T17:46:22.047 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39155 2024-05-04T17:46:22.047 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.047 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.047 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s handler_error 2024-05-04T17:46:22.047 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' handler_error' 2024-05-04T17:46:22.047 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.047 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-05-04T17:46:22.047 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.047 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.047 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.048 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout:ndle_auth_reply_more payload 9 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.685+0000 7f3e37088640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.685+0000 7f3e37088640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.686+0000 7f3e37889640 10 monclient(hunting): handle_auth_done global_id 4386 payload 824 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.686+0000 7f3e37889640 1 -- 172.21.15.19:0/1329173258 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x55dfbde3b590 msgr2=0x55dfbdfae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.686+0000 7f3e37889640 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x55dfbde3b590 0x55dfbdfae6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.686+0000 7f3e37889640 1 -- 172.21.15.19:0/1329173258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfbde50b70 msgr2=0x55dfbdeede00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.686+0000 7f3e37889640 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfbde50b70 0x55dfbdeede00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.686+0000 7f3e37889640 10 monclient: _finish_hunting 0 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.686+0000 7f3e37889640 1 monclient: found mon.noname-a 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.686+0000 7f3e37889640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.686+0000 7f3e37889640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.686+0000 7f3e37889640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfbdfb0de0 con 0x55dfbde37740 2024-05-04T17:46:22.049 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.699+0000 7f3e37889640 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55dfbde37740 0x55dfbdfb06b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3e28005cf0 tx=0x7f3e28053530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.699+0000 7f3e35084640 10 client.?.objecter ms_handle_connect 0x55dfbde37740 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.699+0000 7f3e35084640 10 client.?.objecter resend_mon_ops 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e28061020 con 0x55dfbde37740 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 10 monclient: dump: 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: epoch 1 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: min_mon_release 19 (squid) 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: election_strategy: 1 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:46:22.050 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:46:22.051 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.051 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 10 monclient: _finish_auth 0 2024-05-04T17:46:22.054 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.054 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.054 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.054 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.054 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-05-04T17:46:22.054 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.054 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.054 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.054 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.054 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.055 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo handler_error 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=handler_error 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39180 2024-05-04T17:46:22.056 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test handler_error on pid 39180' 2024-05-04T17:46:22.057 INFO:tasks.workunit.client.0.smithi019.stdout: atest handler_error on pid 39180 2024-05-04T17:46:22.072 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39180 2024-05-04T17:46:22.072 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.072 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.072 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.072 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.072 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.072 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.072 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.072 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.072 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.073 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.074 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.075 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.076 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s io 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' io' 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.077 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.078 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.079 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.080 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo io 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.081 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.082 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39145/exe 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.083 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=io 2024-05-04T17:46:22.084 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39254 2024-05-04T17:46:22.084 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test io on pid 39254' 2024-05-04T17:46:22.084 INFO:tasks.workunit.client.0.smithi019.stdout:test io on pid 39254 2024-05-04T17:46:22.096 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39254 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.097 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.098 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39088/exe 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s ec_io 2024-05-04T17:46:22.099 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' ec_io' 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.100 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.101 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.102 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.103 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.104 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo ec_io 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_cls 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_cls.log 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ cls: /' 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.105 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.106 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=ec_io 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39321 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.107 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.108 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.108 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.108 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.108 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.108 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-05-04T17:46:22.108 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.108 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.108 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.108 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test ec_io on pid 39321' 2024-05-04T17:46:22.108 INFO:tasks.workunit.client.0.smithi019.stdout:test ec_io on pid 39321 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39321 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.114 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s list 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' list' 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39180/exe 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.115 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.116 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo list 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39155/exe 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.117 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.118 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=list 2024-05-04T17:46:22.118 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.118 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.118 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.118 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39337 2024-05-04T17:46:22.118 INFO:tasks.workunit.client.0.smithi019.stdout:test list on pid 39337 2024-05-04T17:46:22.130 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test list on pid 39337' 2024-05-04T17:46:22.130 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39337 2024-05-04T17:46:22.130 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.130 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.130 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_handler_error 2024-05-04T17:46:22.130 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-05-04T17:46:22.130 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.130 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.130 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.130 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.130 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.130 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.131 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s ec_list 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_cmd 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.132 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ cmd: /' 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_cmd.log 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' ec_list' 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.133 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_handler_error.log 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ handler_error: /' 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.134 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.135 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.136 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo ec_list 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.137 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.138 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.139 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.139 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.139 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.139 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.139 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.139 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.139 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.139 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=ec_list 2024-05-04T17:46:22.139 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39375 2024-05-04T17:46:22.139 INFO:tasks.workunit.client.0.smithi019.stdout:test ec_list on pid 39375 2024-05-04T17:46:22.152 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test ec_list on pid 39375' 2024-05-04T17:46:22.152 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39375 2024-05-04T17:46:22.152 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.152 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.152 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-05-04T17:46:22.152 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.152 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.153 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39254/exe 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.154 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.155 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.156 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_io.log 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_io 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ io: /' 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.157 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.158 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s misc 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' misc' 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.159 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo misc 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.160 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.161 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=misc 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39421 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.162 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.163 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test misc on pid 39421' 2024-05-04T17:46:22.163 INFO:tasks.workunit.client.0.smithi019.stdout:test misc on pid 39421 2024-05-04T17:46:22.169 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39421 2024-05-04T17:46:22.169 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.169 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.169 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.169 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.169 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s pool 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' pool' 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.170 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.171 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo pool 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39321/exe 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.172 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.173 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.174 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.175 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.175 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.175 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=pool 2024-05-04T17:46:22.175 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39441 2024-05-04T17:46:22.175 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test pool on pid 39441' 2024-05-04T17:46:22.175 INFO:tasks.workunit.client.0.smithi019.stdout:test pool on pid 39441 2024-05-04T17:46:22.185 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.185 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.185 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.185 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39441 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.186 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.187 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.188 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ ec_io: /' 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.189 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_ec_io 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.190 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s read_operations 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_ec_io.log 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.191 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' read_operations' 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.192 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo read_operations 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39375/exe 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.193 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39337/exe 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_ec_list.log 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.194 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=read_operations 2024-05-04T17:46:22.195 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.195 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39488 2024-05-04T17:46:22.195 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test read_operations on pid 39488' 2024-05-04T17:46:22.195 INFO:tasks.workunit.client.0.smithi019.stdout:test read_operations on pid 39488 2024-05-04T17:46:22.198 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39488 2024-05-04T17:46:22.198 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.198 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.198 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-05-04T17:46:22.198 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.198 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.198 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s snapshots 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' snapshots' 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.199 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_ec_list 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.200 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout:pi_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 10 monclient: _check_auth_tickets 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e28053870 con 0x55dfbde37740 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 10 monclient: handle_config config(0 keys) 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e28058040 con 0x55dfbde37740 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35084640 10 monclient: dump: 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: epoch 1 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: created 2024-05-04T17:43:32.318245+0000 2024-05-04T17:46:22.201 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: min_mon_release 19 (squid) 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: election_strategy: 1 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e39d86780 5 monclient: authenticate success, global_id 4386 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e39d86780 10 monclient: _renew_subs 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e39d86780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55dfbde6fe40 con 0x55dfbde37740 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e35885640 4 set_mon_vals no callback set 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e39d86780 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e39d86780 10 monclient: _renew_subs 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e39d86780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:46:22.202 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfbde70400 con 0x55dfbde37740 2024-05-04T17:46:22.203 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e39d86780 1 librados: init done 2024-05-04T17:46:22.203 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.863+0000 7f3e39d86780 10 librados: wait_for_osdmap waiting 2024-05-04T17:46:22.203 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.996+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3e28064020 con 0x55dfbde37740 2024-05-04T17:46:22.203 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.997+0000 7f3e35084640 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f3e2406ba30 0x7f3e2406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:46:22.203 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.997+0000 7f3e37088640 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f3e2406ba30 0x7f3e2406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:46:22.207 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.207 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.207 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.207 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.207 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.207 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_list.log 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ ec_list: /' 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.208 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.209 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo snapshots 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ list: /' 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_list 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.210 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.211 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.212 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.212 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.212 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.212 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=snapshots 2024-05-04T17:46:22.212 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.212 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39514 2024-05-04T17:46:22.212 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test snapshots on pid 39514' 2024-05-04T17:46:22.212 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.997+test snapshots on pid 39514 2024-05-04T17:46:22.226 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39514 2024-05-04T17:46:22.226 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.227 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.228 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.229 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.230 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39421/exe 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.231 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.232 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.233 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ misc: /' 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.234 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s watch_notify 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' watch_notify' 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_misc.log 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.235 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo watch_notify 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_misc 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.236 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=watch_notify 2024-05-04T17:46:22.237 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39539 2024-05-04T17:46:22.237 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test watch_notify on pid 39539' 2024-05-04T17:46:22.237 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-05-04T17:46:22.237 INFO:tasks.workunit.client.0.smithi019.stdout:test watch_notify on pid 39539 2024-05-04T17:46:22.240 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39539 2024-05-04T17:46:22.240 INFO:tasks.workunit.client.0.smithi019.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-05-04T17:46:22.240 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.240 INFO:tasks.workunit.client.0.smithi019.stderr:++ printf %25s write_operations 2024-05-04T17:46:22.240 INFO:tasks.workunit.client.0.smithi019.stderr:+ r=' write_operations' 2024-05-04T17:46:22.240 INFO:tasks.workunit.client.0.smithi019.stderr:++ awk '{print $1}' 2024-05-04T17:46:22.240 INFO:tasks.workunit.client.0.smithi019.stderr:++ echo write_operations 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.241 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:+ ff=write_operations 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39550 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:+ echo 'test write_operations on pid 39550' 2024-05-04T17:46:22.242 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39441/exe 2024-05-04T17:46:22.243 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.243 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.243 INFO:tasks.workunit.client.0.smithi019.stdout:test write_operations on pid 39550 2024-05-04T17:46:22.268 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.268 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.268 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.268 INFO:tasks.workunit.client.0.smithi019.stderr:+ pids[$f]=39550 2024-05-04T17:46:22.268 INFO:tasks.workunit.client.0.smithi019.stderr:+ ret=0 2024-05-04T17:46:22.268 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' 1 -eq 1 ']' 2024-05-04T17:46:22.268 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38492 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38492 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.269 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -f /etc/bashrc ']' 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:+ . /etc/bashrc 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -z '' ']' 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:++ BASHRCSOURCED=Y 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.270 INFO:tasks.workunit.client.0.smithi019.stderr:++ shopt -q login_shell 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:+++ umask 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.271 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' 0022 -eq 0 ']' 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:++ SHELL=/bin/bash 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/bash_completion.sh 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.272 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/cobbler.sh 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.273 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorgrep.sh 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39488/exe 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_pool 2024-05-04T17:46:22.274 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.275 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.276 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'grep=grep --color=auto' 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'egrep=egrep --color=auto' 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorls.sh 2024-05-04T17:46:22.277 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' '!' -t 0 ']' 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:+++ return 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export S_COLORS=auto 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:+++ S_COLORS=auto 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.278 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_pool.log 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ read_operations: /' 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ pool: /' 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.279 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_read_operations 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.280 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.281 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:+++ /usr/libexec/grepconf.sh -c 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39514/exe 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.282 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.283 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_snapshots 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_snapshots.log 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ snapshots: /' 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_read_operations.log 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.284 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/debuginfod.sh 2024-05-04T17:46:22.285 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.286 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/gawk.sh 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/lang.sh 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39550/exe 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.287 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/locale 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_write_operations 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_write_operations.log 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ write_operations: /' 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG_backup=en_US.UTF-8 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /etc/locale.conf ']' 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -x /usr/bin/sed ']' 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:++++ export LANG=en_US.UTF-8 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:++++ LANG=en_US.UTF-8 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-05-04T17:46:22.288 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:+++ LANG=en_US.UTF-8 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:+++ unset LANG_backup config 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n '' ']' 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' dumb = linux ']' 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/less.sh 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:++ for i in /etc/profile.d/*.sh 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:++ '[' '' ']' 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:++ . /etc/profile.d/which2.sh 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:+++++ readlink /proc/39539/exe 2024-05-04T17:46:22.289 INFO:tasks.workunit.client.0.smithi019.stderr:++++ basename /usr/bin/bash 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_declare='declare -f' 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:+++ which_opt=-f 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export which_declare 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:+++ export -f which 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset i 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:++ unset -f pathmunge 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:+ export PATH 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:+ unset rc 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:+ ceph_test_neorados_watch_notify 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:+ sed 's/^/ watch_notify: /' 2024-05-04T17:46:22.290 INFO:tasks.workunit.client.0.smithi019.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-05-04T17:46:24.513 INFO:tasks.workunit.client.0.smithi019.stdout:0000 7f3e37088640 10 monclient: get_auth_request con 0x7f3e2406ba30 auth_method 0 2024-05-04T17:46:24.513 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.998+0000 7f3e37088640 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f3e2406ba30 0x7f3e2406def0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3e30006fd0 tx=0x7f3e30008040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:46:24.513 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:21.998+0000 7f3e35084640 10 client.4386.objecter ms_handle_connect 0x7f3e2406ba30 2024-05-04T17:46:24.513 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.002+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5210+0+0 (secure 0 0 0) 0x7f3e28097480 con 0x55dfbde37740 2024-05-04T17:46:24.513 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.002+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(25..25 src has 1..25) 2024-05-04T17:46:24.513 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.002+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [25,25] > 0 2024-05-04T17:46:24.513 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.002+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding full epoch 25 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.002+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.002+0000 7f3e39d86780 10 librados: wait_for_osdmap done waiting 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.002+0000 7f3e39d86780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.002+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3e2c00e670 con 0x55dfbde37740 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.120+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3e28095680 con 0x55dfbde37740 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.120+0000 7f3e35084640 10 monclient: handle_get_version_reply finishing 1 version 25 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.120+0000 7f3e36086640 10 client.4386.objecter _get_latest_version latest 25, have it 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.120+0000 7f3e39d86780 10 client.4386.objecter create_pool name=ceph_test_rados_api_asio 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.120+0000 7f3e39d86780 10 client.4386.objecter pool_op_submit 1 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.120+0000 7f3e39d86780 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:22.120+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x55dfbddef600 con 0x55dfbde37740 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.511+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v27) ==== 43+0+0 (secure 0 0 0) 0x7f3e2806b030 con 0x55dfbde37740 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.511+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 pool_op_reply(tid 1 (0) Success v27) 2024-05-04T17:46:24.514 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.511+0000 7f3e35084640 10 client.4386.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v27) 2024-05-04T17:46:24.515 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.511+0000 7f3e35084640 10 client.4386.objecter have request 1 at 0x55dfbde70010 Op: create 2024-05-04T17:46:24.515 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.511+0000 7f3e35084640 20 client.4386.objecter waiting for client to reach epoch 27 before calling back 2024-05-04T17:46:24.515 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.511+0000 7f3e35084640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:46:24.515 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.511+0000 7f3e35084640 10 monclient: _renew_subs 2024-05-04T17:46:24.845 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.511+0000 7f3e35084640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:46:24.845 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.511+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=26}) -- 0x7f3e240717a0 con 0x55dfbde37740 2024-05-04T17:46:24.845 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.511+0000 7f3e35084640 10 client.4386.objecter done 2024-05-04T17:46:24.845 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.637+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:24.845 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.637+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:24.845 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.637+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:24.845 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.637+0000 7f3e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.842+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 8 ==== osd_map(26..27 src has 1..27) ==== 16916+0+0 (secure 0 0 0) 0x7f3e280976e0 con 0x55dfbde37740 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(26..27 src has 1..27) 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [26,27] > 25 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 26 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 27 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 10 librados: set snap read head -> 1 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 10 librados: writefull oid=exist nspace= 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfaec20 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 27 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x55dfbdfb15a0 0x55dfbdee7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:46:24.846 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [writefull 0~5 in=5b] tid 2 osd.7 2024-05-04T17:46:24.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:46:24.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 2 2024-05-04T17:46:24.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 15 client.4386.objecter _send_op 2 to 14.1 on osd.7 2024-05-04T17:46:24.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x55dfbdee7550 con 0x55dfbdfb15a0 2024-05-04T17:46:24.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:46:33.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:24.843+0000 7f3e39d86780 5 client. api_service: [==========] Running 4 tests from 1 test suite. 2024-05-04T17:46:33.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [----------] Global test environment set-up. 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [----------] 4 tests from LibRadosService 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5410 ms) 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [ OK ] LibRadosService.RegisterLate (56 ms) 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: cluster: 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: id: 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: health: HEALTH_WARN 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 23 pool(s) do not have an application enabled 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: services: 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-05-04T17:46:33.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: osd: 8 osds: 8 up (since 73s), 8 in (since 2m) 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: data: 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: pools: 35 pools, 1065 pgs 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: objects: 2 objects, 449 KiB 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: pgs: 88.169% pgs unknown 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 10.986% pgs not active 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 939 unknown 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 82 creating+peering 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 35 creating+activating 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 9 active+clean 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:33.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: cluster: 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: id: 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: health: HEALTH_WARN 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 16 pool(s) do not have an application enabled 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: services: 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: osd: 8 osds: 8 up (since 75s), 8 in (since 2m) 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: foo: 15 portals active (1 hosts, 3 zones) 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: data: 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: pools: 36 pools, 1025 pgs 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: objects: 2 objects, 449 KiB 2024-05-04T17:46:33.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: pgs: 53.659% pgs unknown 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 40.780% pgs not active 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 550 unknown 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 354 creating+peering 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 64 creating+activating 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 57 active+clean 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: cluster: 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: id: 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: health: HEALTH_WARN 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 20 pool(s) do not have an application enabled 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: services: 2024-05-04T17:46:33.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-05-04T17:46:33.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-05-04T17:46:33.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: osd: 8 osds: 8 up (since 77s), 8 in (since 2m) 2024-05-04T17:46:33.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-05-04T17:46:33.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-05-04T17:46:33.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:33.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: data: 2024-05-04T17:46:33.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: pools: 39 pools, 1121 pgs 2024-05-04T17:46:34.641 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: objects: 2 objects, 444386.objecter 1 in flight 2024-05-04T17:46:34.641 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:25.268+0000 7f3e3808a640 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x55dfbdfb15a0 0x55dfbdee7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T17:46:34.641 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:25.268+0000 7f3e3808a640 10 monclient: get_auth_request con 0x55dfbdfb15a0 auth_method 0 2024-05-04T17:46:34.641 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:25.573+0000 7f3e35084640 10 client.4386.objecter ms_handle_connect 0x55dfbdfb15a0 2024-05-04T17:46:34.641 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:25.573+0000 7f3e3808a640 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x55dfbdfb15a0 0x55dfbdee7010 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T17:46:34.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:25.637+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:34.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:25.637+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:34.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:25.637+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:34.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:26.639+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:34.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:26.639+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:34.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:26.639+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:34.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:26.864+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:46:34.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:27.639+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:34.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:27.639+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:34.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:27.640+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:34.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:28.640+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:34.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:28.640+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:34.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:28.640+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:34.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:29.640+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:34.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:29.640+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:34.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:29.640+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:34.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:30.640+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:34.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:30.640+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:34.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:30.640+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:34.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:31.640+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:34.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:31.640+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:34.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:31.640+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:34.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:31.865+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:46:34.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:32.640+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:34.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:32.640+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:34.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:32.640+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:34.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:33.640+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:34.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:33.640+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:34.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:33.640+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:35.524 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:30.455 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:35.629 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:30.518 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:35.738 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:30.669 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:35.769 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:30.700 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:35.816 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:30.705 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:35.817 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:30.748 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:35.936 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:30.867 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:36.035 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:30.966 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:36.234 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:31.123 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:37.505 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:34.640+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:37.505 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:34.640+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:37.505 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:34.640+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:37.505 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:34.640+0000 7f3e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:46:37.505 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:35.640+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:37.505 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:35.640+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:37.505 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:35.640+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:37.505 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:36.641+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:37.505 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:36.641+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:37.505 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:36.641+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:36.866+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:36.866+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:36.866+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:36.866+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:36.866+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:36.866+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=28}) -- 0x7f3e08002f90 con 0x55dfbde37740 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:36.866+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e08003360 con 0x55dfbdfb15a0 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.503+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 9 ==== osd_map(28..35 src has 1..35) ==== 16092+0+0 (secure 0 0 0) 0x7f3e280993a0 con 0x55dfbde37740 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.503+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(28..35 src has 1..35) 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [28,35] > 27 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 28 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 28 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:37.506 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:37.507 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 29 2024-05-04T17:46:37.507 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:37.507 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 29 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:37.507 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:37.507 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 30 2024-05-04T17:46:37.507 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:39.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 30 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:39.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:39.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 31 2024-05-04T17:46:39.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:39.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 31 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:39.642 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:39.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 32 2024-05-04T17:46:39.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:39.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 32 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:39.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:39.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 33 2024-05-04T17:46:39.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:39.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 33 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:39.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:39.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 34 2024-05-04T17:46:39.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:39.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 34 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:39.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:39.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 35 2024-05-04T17:46:39.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:39.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 35 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:39.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:39.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:46:39.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.504+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:46:39.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.641+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:39.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.641+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:39.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:37.641+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:39.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:38.641+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:39.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:38.641+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:39.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:38.641+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:39.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:39.641+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:39.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:39.641+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:41.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:39.641+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:41.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:40.641+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:41.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:40.641+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:41.872 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:40.641+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:41.872 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.641+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:41.872 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.641+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:41.872 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.641+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:41.872 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.867+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:46:41.872 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.867+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.867+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.867+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.867+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.867+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=36}) -- 0x7f3e08003560 con 0x55dfbde37740 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.867+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e080037f0 con 0x55dfbdfb15a0 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 10 ==== osd_map(36..38 src has 1..38) ==== 4246+0+0 (secure 0 0 0) 0x7f3e280c92a0 con 0x55dfbde37740 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(36..38 src has 1..38) 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [36,38] > 35 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 36 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 36 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:41.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 37 2024-05-04T17:46:41.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:41.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 37 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:41.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:41.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 38 2024-05-04T17:46:41.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:41.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 38 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:41.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:41.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:46:47.065 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:41.870+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:46:47.065 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:42.641+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:47.065 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:42.641+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:47.065 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:42.641+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:47.065 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:43.641+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:47.065 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:43.641+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:47.065 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:43.641+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:47.065 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:44.641+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:47.065 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:44.641+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:47.065 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:44.641+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:44.641+0000 7f3e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:45.642+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:45.642+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:45.642+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:46.642+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:46.642+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:46.642+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:46.868+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:46.868+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:46.868+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:46.868+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:46.868+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:46:47.066 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:46.868+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=39}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:46:47.067 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:46.868+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e08003ca0 con 0x55dfbdfb15a0 2024-05-04T17:46:47.067 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.063+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 11 ==== osd_map(39..41 src has 1..41) ==== 1575+0+0 (secure 0 0 0) 0x7f3e280b8330 con 0x55dfbde37740 2024-05-04T17:46:47.067 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(39..41 src has 1..41) 2024-05-04T17:46:47.067 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [39,41] > 38 2024-05-04T17:46:47.067 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 39 2024-05-04T17:46:47.067 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:47.067 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 39 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:47.067 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 4 pool: Running main() from gmock_main.cc 2024-05-04T17:46:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-05-04T17:46:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [----------] Global test environment set-up. 2024-05-04T17:46:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-05-04T17:46:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-05-04T17:46:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ OK ] NeoRadosPools.PoolList (4487 ms) 2024-05-04T17:46:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-05-04T17:46:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (6231 ms) 2024-05-04T17:46:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-05-04T17:46:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (3160 ms) 2024-05-04T17:46:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-05-04T17:46:49.917 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (5778 ms) 2024-05-04T17:46:49.917 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-05-04T17:46:49.917 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (5068 ms) 2024-05-04T17:46:49.917 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-05-04T17:46:49.917 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (3133 ms) 2024-05-04T17:46:49.917 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [----------] 6 tests from NeoRadosPools (27857 ms total) 2024-05-04T17:46:49.917 INFO:tasks.workunit.client.0.smithi019.stdout: pool: 2024-05-04T17:46:49.917 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [----------] Global test environment tear-down 2024-05-04T17:46:49.917 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [==========] 6 tests from 1 test suite ran. (27857 ms total) 2024-05-04T17:46:49.917 INFO:tasks.workunit.client.0.smithi019.stdout: pool: [ PASSED ] 6 tests. 2024-05-04T17:46:52.643 INFO:tasks.workunit.client.0.smithi019.stdout:0 2024-05-04T17:46:52.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:52.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 40 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:52.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:52.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 41 2024-05-04T17:46:52.643 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 41 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.064+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.642+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.642+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:47.642+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:48.642+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:48.642+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:48.642+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:49.642+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:49.642+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:49.642+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:52.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:50.642+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:50.642+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:50.642+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:51.642+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:51.642+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:51.642+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:51.869+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:51.869+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:51.869+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:51.869+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:51.869+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:51.869+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=42}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:51.869+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e08003ec0 con 0x55dfbdfb15a0 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:52.642+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:52.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:52.642+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:53.742 INFO:tasks.workunit.client.0.smithi019.stdout: api_as9 KiB 2024-05-04T17:46:53.742 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: pgs: 49.688% pgs unknown 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 44.781% pgs not active 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 557 unknown 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 421 creating+peering 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 81 creating+activating 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 61 active+clean 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 1 active 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [ OK ] LibRadosService.StatusFormat (6367 ms) 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [ RUN ] LibRadosService.Status 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [ OK ] LibRadosService.Status (20059 ms) 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [----------] 4 tests from LibRadosService (31892 ms total) 2024-05-04T17:46:53.743 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: 2024-05-04T17:46:53.744 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [----------] Global test environment tear-down 2024-05-04T17:46:53.744 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31892 ms total) 2024-05-04T17:46:53.744 INFO:tasks.workunit.client.0.smithi019.stdout: api_service: [ PASSED ] 4 tests. 2024-05-04T17:46:54.282 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-05-04T17:46:54.282 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [----------] Global test environment set-up. 2024-05-04T17:46:54.282 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-05-04T17:46:54.282 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5642 ms) 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (684 ms) 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20533 ms) 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: attempt 0 of 20 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: failed to deregister: 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: { 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "epoch": 13, 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "modified": "2024-05-04T17:46:48.867197+0000", 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "services": { 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "laundry": { 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "daemons": { 2024-05-04T17:46:54.283 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "summary": "", 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "close-test-pid38976": { 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "start_epoch": 13, 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "start_stamp": "2024-05-04T17:46:48.779531+0000", 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "gid": 4786, 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "addr": "172.21.15.19:0/3842216095", 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "metadata": { 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "arch": "x86_64", 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "ceph_release": "squid", 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-2497-gf273489c (f273489c6dcd4bc88409993babd09dd99491162c) squid (dev)", 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "ceph_version_short": "19.0.0-2497-gf273489c", 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "distro": "centos", 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "distro_version": "9", 2024-05-04T17:46:54.284 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "foo": "bar", 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "hostname": "smithi019", 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024", 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "kernel_version": "5.14.0-437.el9.x86_64", 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "mem_swap_kb": "0", 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "mem_total_kb": "32494644", 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "os": "Linux", 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "this": "that" 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: }, 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "task_status": {} 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: }, 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "pid39036": { 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "start_epoch": 12, 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "start_stamp": "2024-05-04T17:46:33.765989+0000", 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "gid": 4553, 2024-05-04T17:46:54.285 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "addr": "172.21.15.19:0/3781475755", 2024-05-04T17:46:54.286 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "metadata": { 2024-05-04T17:46:54.286 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "arch": "x86_64", 2024-05-04T17:46:54.286 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "ceph_release": "squid", 2024-05-04T17:46:54.286 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-2497-gf273489c (f273489c6dcd4bc88409993babd09dd99491162c) squid (dev)", 2024-05-04T17:46:54.286 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "ceph_version_short": "19.0.0-2497-gf273489c", 2024-05-04T17:46:54.286 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-05-04T17:46:54.918 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:46:54.771+0000 ea5d640 -1 osd.7 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4440.0:2 19.1a 19:5c6776db:::0.obj:head [write 0~189 in=189b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:54.935 INFO:tasks.workunit.client.0.smithi019.stdout: io: 2024-05-04T17:46:52.642+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:54.935 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.196+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 12 ==== osd_map(42..44 src has 1..44) ==== 1536+0+0 (secure 0 0 0) 0x7f3e28095a50 con 0x55dfbde37740 2024-05-04T17:46:54.935 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.196+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(42..44 src has 1..44) 2024-05-04T17:46:54.935 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.196+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [42,44] > 41 2024-05-04T17:46:54.935 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.196+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 42 2024-05-04T17:46:54.935 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:54.935 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 42 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:54.935 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:54.935 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 43 2024-05-04T17:46:54.935 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:54.935 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 43 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 44 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 44 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.197+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.642+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.642+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:53.642+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:54.642+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:54.642+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:54.642+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:54.642+0000 7f3e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:46:54.936 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:54.934+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 1 ==== osd_map(28..43 src has 1..43) ==== 23081+0+0 (crc 0 0 0) 0x7f3e2c00e470 con 0x55dfbdfb15a0 2024-05-04T17:46:54.937 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:54.934+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(28..43 src has 1..43) 2024-05-04T17:46:54.937 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:54.934+0000 7f3e35084640 3 client.4386.objecter handle_osd_map ignoring epochs [28,43] <= 44 2024-05-04T17:46:54.937 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:54.934+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:46:55.260 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:46:55.209+0000 12a65640 -1 osd.4 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:55.675 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:46:55.662+0000 ea5d640 -1 osd.2 44 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4486.0:2 8.1b 8:dbdb5272:test-rados-api-smithi019-38568-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:55.779 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:50.710 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:55.790 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:50.720 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:55.860 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:50.791 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:55.862 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:50.793 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:55.862 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:50.752 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:55.868 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:50.757 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:56.031 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:02:50.920 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:56.048 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:02:50.979 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:56.094 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:46:56.084+0000 ea5d640 -1 osd.7 45 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4440.0:2 19.1a 19:5c6776db:::0.obj:head [write 0~189 in=189b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:56.185 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:02:51.115 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:46:56.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:46:56.639+0000 12a65640 -1 osd.4 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:54.934+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:55.642+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:55.642+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:55.642+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.643+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.643+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.643+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.870+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.870+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.870+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.870+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.870+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:46:56.892 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.870+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=45}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.870+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e080040f0 con 0x55dfbdfb15a0 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 13 ==== osd_map(45..47 src has 1..47) ==== 1957+0+0 (secure 0 0 0) 0x7f3e28095a50 con 0x55dfbde37740 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(45..47 src has 1..47) 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [45,47] > 44 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 45 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 45 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 46 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 46 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 47 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 47 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:46:56.893 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:46:56.894 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:46:57.095 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:46:56.899+0000 ea5d640 -1 osd.2 46 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4486.0:2 8.1b 8:dbdb5272:test-rados-api-smithi019-38568-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:57.142 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:46:56.924+0000 12a65640 -1 osd.1 46 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4458.0:2 25.0 25.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:57.323 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:46:57.310+0000 ea5d640 -1 osd.7 46 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4440.0:2 19.1a 19:5c6776db:::0.obj:head [write 0~189 in=189b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:57.986 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:46:57.981+0000 ea5d640 -1 osd.2 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4486.0:2 8.1b 8:dbdb5272:test-rados-api-smithi019-38568-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:58.200 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:46:58.129+0000 12a65640 -1 osd.1 46 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4458.0:2 25.0 25:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:58.415 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:46:58.372+0000 ea5d640 -1 osd.7 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4440.0:2 19.1a 19:5c6776db:::0.obj:head [write 0~189 in=189b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:58.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:46:58.417+0000 12a65640 -1 osd.4 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:59.067 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:46:59.023+0000 ea5d640 -1 osd.2 47 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4486.0:2 8.1b 8:dbdb5272:test-rados-api-smithi019-38568-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:59.164 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:46:59.159+0000 12a65640 -1 osd.1 46 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4458.0:2 25.0 25:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:46:59.916 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:46:59.903+0000 ea5d640 -1 osd.7 48 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4440.0:2 19.1a 19:5c6776db:::0.obj:head [write 0~189 in=189b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:00.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:00.259+0000 12a65640 -1 osd.4 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:00.788 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:00.570+0000 ea5d640 -1 osd.2 49 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4486.0:2 8.1b 8:dbdb5272:test-rados-api-smithi019-38568-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:00.932 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:00.881+0000 ea5d640 -1 osd.7 49 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4440.0:2 19.1a 19:5c6776db:::0.obj:head [write 0~189 in=189b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:01.062 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:01.048+0000 12a65640 -1 osd.6 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:01.348 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:47:01.295+0000 12a65640 -1 osd.1 47 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4458.0:2 25.0 25:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:01.699 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:01.648+0000 ea5d640 -1 osd.2 49 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4486.0:2 8.1b 8:dbdb5272:test-rados-api-smithi019-38568-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:01.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:56.891+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:47:01.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:57.643+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:01.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:57.643+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:01.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:57.643+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:01.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:58.643+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:01.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:58.643+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:01.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:58.643+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:01.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:59.643+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:59.643+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:46:59.643+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:00.643+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:00.643+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:00.643+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.643+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.643+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.643+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.871+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.871+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.871+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.871+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.871+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:47:01.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.871+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=48}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:47:01.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.871+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e080040f0 con 0x55dfbdfb15a0 2024-05-04T17:47:01.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 14 ==== osd_map(48..51 src has 1..51) ==== 3955+0+0 (secure 0 0 0) 0x7f3e28095a50 con 0x55dfbde37740 2024-05-04T17:47:01.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(48..51 src has 1..51) 2024-05-04T17:47:01.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [48,51] > 47 2024-05-04T17:47:01.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 48 2024-05-04T17:47:01.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:01.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 48 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:01.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:01.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 49 2024-05-04T17:47:01.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:01.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:01.983+0000 12a65640 -1 osd.4 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:02.215 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:02.205+0000 ea5d640 -1 osd.7 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4440.0:2 19.1a 19:5c6776db:::0.obj:head [write 0~189 in=189b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:02.592 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:02.585+0000 12a65640 -1 osd.6 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:02.776 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:02.729+0000 ea5d640 -1 osd.2 49 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4486.0:2 8.1b 8:dbdb5272:test-rados-api-smithi019-38568-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:03.311 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:03.274+0000 12a65640 -1 osd.4 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:03.335 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:47:03.202+0000 12a65640 -1 osd.1 48 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4458.0:2 25.0 25:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:03.438 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:03.319+0000 ea5d640 -1 osd.7 52 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4440.0:2 19.1a 19:5c6776db:::0.obj:head [write 0~189 in=189b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:03.717 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:03.710+0000 12a65640 -1 osd.6 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:04.170 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:03.974+0000 ea5d640 -1 osd.2 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4412.0:2 30.6 30:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:04.213 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 20 client.4386.objecter _ca handler_error: Running main() from gmock_main.cc 2024-05-04T17:47:04.213 INFO:tasks.workunit.client.0.smithi019.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-05-04T17:47:04.213 INFO:tasks.workunit.client.0.smithi019.stdout: handler_error: [----------] Global test environment set-up. 2024-05-04T17:47:04.213 INFO:tasks.workunit.client.0.smithi019.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-05-04T17:47:04.213 INFO:tasks.workunit.client.0.smithi019.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-05-04T17:47:04.213 INFO:tasks.workunit.client.0.smithi019.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (42258 ms) 2024-05-04T17:47:04.213 INFO:tasks.workunit.client.0.smithi019.stdout: handler_error: [----------] 1 test from neocls_handler_error (42258 ms total) 2024-05-04T17:47:04.213 INFO:tasks.workunit.client.0.smithi019.stdout: handler_error: 2024-05-04T17:47:04.213 INFO:tasks.workunit.client.0.smithi019.stdout: handler_error: [----------] Global test environment tear-down 2024-05-04T17:47:04.213 INFO:tasks.workunit.client.0.smithi019.stdout: handler_error: [==========] 1 test from 1 test suite ran. (42258 ms total) 2024-05-04T17:47:04.213 INFO:tasks.workunit.client.0.smithi019.stdout: handler_error: [ PASSED ] 1 test. 2024-05-04T17:47:04.495 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:04.474+0000 ea5d640 -1 osd.7 52 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4440.0:2 19.1a 19:5c6776db:::0.obj:head [write 0~189 in=189b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:04.554 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:04.526+0000 12a65640 -1 osd.4 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:04.624 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:47:04.428+0000 12a65640 -1 osd.1 49 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4464.0:2 26.0 26:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:04.737 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "distro": "centos", 2024-05-04T17:47:04.738 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-05-04T17:47:04.738 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "distro_version": "9", 2024-05-04T17:47:04.738 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "foo": "bar", 2024-05-04T17:47:04.738 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "hostname": "smithi019", 2024-05-04T17:47:04.738 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024", 2024-05-04T17:47:04.738 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "kernel_version": "5.14.0-437.el9.x86_64", 2024-05-04T17:47:04.738 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "mem_swap_kb": "0", 2024-05-04T17:47:04.738 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "mem_total_kb": "32494644", 2024-05-04T17:47:04.738 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "os": "Linux", 2024-05-04T17:47:04.738 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "this": "that" 2024-05-04T17:47:04.738 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: }, 2024-05-04T17:47:04.738 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "task_status": {} 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: } 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: } 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: } 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: } 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: } 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: attempt 1 of 20 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: failed to deregister: 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: { 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "epoch": 15, 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "modified": "2024-05-04T17:46:54.870357+0000", 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "services": { 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "laundry": { 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "daemons": { 2024-05-04T17:47:04.739 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "summary": "", 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "close-test-pid38976": { 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "start_epoch": 15, 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "start_stamp": "2024-05-04T17:46:54.388001+0000", 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "gid": 4598, 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "addr": "172.21.15.19:0/4148906176", 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "metadata": { 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "arch": "x86_64", 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "ceph_release": "squid", 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-2497-gf273489c (f273489c6dcd4bc88409993babd09dd99491162c) squid (dev)", 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "ceph_version_short": "19.0.0-2497-gf273489c", 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "distro": "centos", 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "distro_version": "9", 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "foo": "bar", 2024-05-04T17:47:04.740 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "hostname": "smithi019", 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024", 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "kernel_version": "5.14.0-437.el9.x86_64", 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "mem_swap_kb": "0", 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "mem_total_kb": "32494644", 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "os": "Linux", 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "this": "that" 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: }, 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: "task_status": {} 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: } 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: } 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: } 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: } 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: } 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: 2024-05-04T17:47:04.741 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: attempt 2 of 20 2024-05-04T17:47:04.742 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (16061 ms) 2024-05-04T17:47:04.742 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (42920 ms total) 2024-05-04T17:47:04.742 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: 2024-05-04T17:47:04.742 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [----------] Global test environment tear-down 2024-05-04T17:47:04.742 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (42920 ms total) 2024-05-04T17:47:04.742 INFO:tasks.workunit.client.0.smithi019.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-05-04T17:47:05.057 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:05.046+0000 ea5d640 -1 osd.2 51 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4412.0:2 30.6 30:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:05.446 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:05.359+0000 12a65640 -1 osd.6 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:05.689 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:47:05.571+0000 12a65640 -1 osd.1 49 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4464.0:2 26.0 26:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:05.750 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:05.667+0000 12a65640 -1 osd.4 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:05.884 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:05.871+0000 ea5d640 -1 osd.7 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:05.900 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:47:05.881+0000 ea5d640 -1 osd.3 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4434.0:2 18.1e 18.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:06.199 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:06.162+0000 ea5d640 -1 osd.2 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4412.0:2 30.6 30:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:06.601 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:06.550+0000 12a65640 -1 osd.6 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:06.873 INFO:tasks.workunit.client.0.smithi019.stdout:lc_target epoch 49 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:06.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:06.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 50 2024-05-04T17:47:06.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:06.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 50 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:06.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:06.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 51 2024-05-04T17:47:06.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:06.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 51 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:06.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:06.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:47:06.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:01.873+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:02.643+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:02.643+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:02.643+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:03.643+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:03.643+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:03.643+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:04.643+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:04.643+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:04.643+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:04.643+0000 7f3e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:05.643+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:05.643+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:05.643+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:06.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.643+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:06.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.643+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:06.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.643+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:06.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.872+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:47:06.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.872+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:47:06.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.872+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:47:06.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.872+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:47:06.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.872+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:47:06.897 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:06.863+0000 12a65640 -1 osd.4 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:06.968 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:47:06.915+0000 12a65640 -1 osd.1 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4464.0:2 26.0 26:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:07.097 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:47:07.090+0000 ea5d640 -1 osd.3 52 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4434.0:2 18.1e 18.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:07.142 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:07.033+0000 ea5d640 -1 osd.7 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:07.232 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:07.227+0000 ea5d640 -1 osd.2 52 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4412.0:2 30.6 30:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:07.729 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:07.659+0000 12a65640 -1 osd.6 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:07.945 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:07.876+0000 12a65640 -1 osd.4 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:08.481 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:47:08.469+0000 12a65640 -1 osd.1 52 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4464.0:2 26.0 26:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:08.525 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:08.516+0000 ea5d640 -1 osd.7 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:08.619 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:08.519+0000 ea5d640 -1 osd.2 53 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4412.0:2 30.6 30:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:08.674 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:47:08.670+0000 ea5d640 -1 osd.3 53 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4434.0:2 18.1e 18.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:08.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:08.721+0000 12a65640 -1 osd.6 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:09.402 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:09.298+0000 12a65640 -1 osd.4 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:09.633 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:09.617+0000 ea5d640 -1 osd.7 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:09.644 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.872+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=52}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.872+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e080040f0 con 0x55dfbdfb15a0 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 15 ==== osd_map(52..55 src has 1..55) ==== 4641+0+0 (secure 0 0 0) 0x7f3e28095a50 con 0x55dfbde37740 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(52..55 src has 1..55) 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [52,55] > 51 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 52 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 52 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 53 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 53 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 54 2024-05-04T17:47:09.645 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 54 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 55 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 55 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:06.875+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:07.643+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:07.644+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:07.644+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:08.644+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:08.644+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:09.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:08.644+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:09.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:09.644+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:09.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:09.644+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:09.953 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:09.871+0000 12a65640 -1 osd.6 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:10.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:10.466+0000 12a65640 -1 osd.4 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:10.687 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:47:10.587+0000 ea5d640 -1 osd.3 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4434.0:2 18.1e 18.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:10.965 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:10.953+0000 ea5d640 -1 osd.7 57 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:11.013 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:10.947+0000 12a65640 -1 osd.6 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:11.142 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:11.132+0000 ea5d640 -1 osd.2 54 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4412.0:2 30.6 30:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:11.515 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:11.509+0000 12a65640 -1 osd.4 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:11.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:09.644+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:11.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:10.644+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:11.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:10.644+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:11.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:10.644+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:11.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.644+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.644+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.644+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.873+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.873+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.873+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.873+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.873+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.873+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=56}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.873+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e08004150 con 0x55dfbdfb15a0 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 16 ==== osd_map(56..59 src has 1..59) ==== 2841+0+0 (secure 0 0 0) 0x7f3e28095a50 con 0x55dfbde37740 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(56..59 src has 1..59) 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [56,59] > 55 2024-05-04T17:47:11.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 56 2024-05-04T17:47:11.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:11.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 56 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:11.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:11.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 57 2024-05-04T17:47:11.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:11.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 57 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:11.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:11.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 58 2024-05-04T17:47:11.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:11.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 58 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:11.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:11.882 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:47:11.869+0000 ea5d640 -1 osd.3 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4434.0:2 18.1e 18.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:12.147 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:12.141+0000 ea5d640 -1 osd.7 58 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:12.191 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:12.100+0000 12a65640 -1 osd.6 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:12.564 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:12.359+0000 ea5d640 -1 osd.2 56 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4412.0:2 30.6 30:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:12.675 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:12.541+0000 12a65640 -1 osd.4 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:13.176 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:13.167+0000 12a65640 -1 osd.6 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:13.214 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:47:13.189+0000 ea5d640 -1 osd.3 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4434.0:2 18.1e 18.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:13.309 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:13.300+0000 ea5d640 -1 osd.7 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:13.423 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:13.413+0000 ea5d640 -1 osd.2 56 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4412.0:2 30.6 30:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:14.327 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:47:14.308+0000 ea5d640 -1 osd.3 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4434.0:2 18.1e 18.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:14.380 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:14.368+0000 12a65640 -1 osd.4 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:14.412 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:14.405+0000 ea5d640 -1 osd.7 61 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:14.522 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:14.465+0000 12a65640 -1 osd.6 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:14.759 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:14.741+0000 ea5d640 -1 osd.2 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4412.0:2 30.6 30:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:15.461 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:15.445+0000 12a65640 -1 osd.4 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:15.604 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:15.537+0000 12a65640 -1 osd.6 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:15.731 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:15.650+0000 ea5d640 -1 osd.7 62 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:16.058 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:47:15.992+0000 ea5d640 -1 osd.3 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4434.0:2 18.1e 18.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:16.082 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:15.981+0000 ea5d640 -1 osd.2 58 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:16.646 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:16.567+0000 12a65640 -1 osd.6 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:16.688 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:16.674+0000 ea5d640 -1 osd.7 62 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:16.761 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:16.621+0000 12a65640 -1 osd.4 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:17.038 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 59 2024-05-04T17:47:17.038 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:17.038 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 59 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:11.877+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:12.644+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:12.644+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:12.644+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:13.644+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:13.644+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:13.644+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:14.644+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:14.644+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:14.644+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:14.644+0000 7f3e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:15.644+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:17.039 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:15.644+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:15.644+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:16.644+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:16.644+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:16.644+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:16.874+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:16.874+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:16.874+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:16.874+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:16.874+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:16.874+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=60}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:16.874+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e08004150 con 0x55dfbdfb15a0 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 17 ==== osd_map(60..63 src has 1..63) ==== 2871+0+0 (secure 0 0 0) 0x7f3e28095a50 con 0x55dfbde37740 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(60..63 src has 1..63) 2024-05-04T17:47:17.040 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [60,63] > 59 2024-05-04T17:47:17.205 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:47:17.162+0000 ea5d640 -1 osd.3 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4434.0:2 18.1e 18:7a7d4169:::src_object.1:head [writefull 0~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:17.560 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:17.452+0000 ea5d640 -1 osd.2 60 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:17.616 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:17.611+0000 12a65640 -1 osd.6 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:17.741 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:17.733+0000 12a65640 -1 osd.4 61 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:17.948 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:17.869+0000 ea5d640 -1 osd.7 64 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:18.783 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:18.774+0000 12a65640 -1 osd.6 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:18.922 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:18.908+0000 ea5d640 -1 osd.7 64 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:19.280 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:19.160+0000 12a65640 -1 osd.4 62 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:19.360 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:19.215+0000 ea5d640 -1 osd.2 62 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:20.142 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:20.081+0000 12a65640 -1 osd.6 61 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:20.250 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:20.242+0000 ea5d640 -1 osd.2 62 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:20.312 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:20.240+0000 ea5d640 -1 osd.7 65 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:20.502 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:20.412+0000 12a65640 -1 osd.4 64 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:21.293 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:16.224 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:21.345 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:16.234 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:21.383 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:16.314 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:21.384 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:16.315 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:21.390 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:16.321 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:21.427 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:21.357+0000 12a65640 -1 osd.6 62 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:21.439 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:21.430+0000 12a65640 -1 osd.4 64 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:21.487 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:21.455+0000 ea5d640 -1 osd.2 63 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:21.530 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:16.419 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:21.697 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:21.686+0000 ea5d640 -1 osd.7 66 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:21.780 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:16.711 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:21.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 60 2024-05-04T17:47:21.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:21.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 60 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:21.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:21.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 61 2024-05-04T17:47:21.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:21.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 61 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:21.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:21.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 62 2024-05-04T17:47:21.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:21.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 62 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:21.876 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 63 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 63 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.037+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.644+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.644+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:17.644+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:18.645+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:18.645+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:18.645+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:21.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:19.645+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:21.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:19.645+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:21.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:19.645+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:21.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:20.645+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:21.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:20.645+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:21.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:20.645+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:21.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.645+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:21.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.645+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:21.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.645+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:21.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.875+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:47:21.890 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:16.779 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:21.924 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: api_misc: [==========] Running 12 tests from 4 test suites. 2024-05-04T17:47:21.924 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [----------] Global test environment set-up. 2024-05-04T17:47:21.924 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-05-04T17:47:21.924 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-05-04T17:47:21.924 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-05-04T17:47:21.924 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-05-04T17:47:21.924 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: 2024-05-04T17:47:21.924 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-05-04T17:47:21.925 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-05-04T17:47:21.925 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: unable to get monitor info from DNS SRV with service name: 2024-05-04T17:46:21.822+0000 7fe6f9840780 -1 failed for service _ceph-mon._tcp 2024-05-04T17:47:21.925 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: ceph-mon 2024-05-04T17:47:21.925 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: 2024-05-04T17:46:21.822+0000 7fe6f9840780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-05-04T17:47:21.925 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (197 ms) 2024-05-04T17:47:21.925 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-05-04T17:47:21.925 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2025 ms) 2024-05-04T17:47:21.925 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2222 ms total) 2024-05-04T17:47:21.925 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: 2024-05-04T17:47:21.926 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-05-04T17:47:21.926 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-05-04T17:47:21.926 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137113bf0 2024-05-04T17:47:21.926 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a136eca660 2024-05-04T17:47:21.926 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137116940 2024-05-04T17:47:21.926 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a136ece700 2024-05-04T17:47:21.926 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a136f80a00 2024-05-04T17:47:21.926 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13724c200 2024-05-04T17:47:21.926 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13724cc20 2024-05-04T17:47:21.926 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13724d690 2024-05-04T17:47:21.926 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13724e0e0 2024-05-04T17:47:21.926 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13724eb30 2024-05-04T17:47:21.927 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13724f580 2024-05-04T17:47:21.927 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137250010 2024-05-04T17:47:21.927 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137250b50 2024-05-04T17:47:21.927 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137251690 2024-05-04T17:47:21.927 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137051230 2024-05-04T17:47:21.927 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137051cc0 2024-05-04T17:47:21.927 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137052800 2024-05-04T17:47:21.927 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137053340 2024-05-04T17:47:21.927 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137053e80 2024-05-04T17:47:21.927 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a1370549c0 2024-05-04T17:47:21.927 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137252150 2024-05-04T17:47:21.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137252bc0 2024-05-04T17:47:21.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137253700 2024-05-04T17:47:21.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137254240 2024-05-04T17:47:21.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137254d80 2024-05-04T17:47:21.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137251d80 2024-05-04T17:47:21.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137251ef0 2024-05-04T17:47:21.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137050e60 2024-05-04T17:47:21.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a1370510c0 2024-05-04T17:47:21.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137253f70 2024-05-04T17:47:21.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137253310 2024-05-04T17:47:21.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137253590 2024-05-04T17:47:21.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137252920 2024-05-04T17:47:21.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a1370545d0 2024-05-04T17:47:21.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137054890 2024-05-04T17:47:21.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137053c00 2024-05-04T17:47:21.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137052ff0 2024-05-04T17:47:21.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137052410 2024-05-04T17:47:21.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137052610 2024-05-04T17:47:21.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137051980 2024-05-04T17:47:21.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a1372512a0 2024-05-04T17:47:21.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137251470 2024-05-04T17:47:21.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a1372549d0 2024-05-04T17:47:21.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13724fc20 2024-05-04T17:47:21.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13724fe80 2024-05-04T17:47:21.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13724f2a0 2024-05-04T17:47:21.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13724e790 2024-05-04T17:47:21.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13724ea20 2024-05-04T17:47:21.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13724de80 2024-05-04T17:47:21.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137027060 2024-05-04T17:47:21.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a1372508b0 2024-05-04T17:47:21.931 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:16.854 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:22.521 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:22.394+0000 12a65640 -1 osd.6 63 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:22.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: starte2024-05-04T17:47:21.875+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:47:22.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.875+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:47:22.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.875+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:47:22.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.875+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:47:22.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.875+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=64}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:47:22.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.875+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e08004150 con 0x55dfbdfb15a0 2024-05-04T17:47:22.646 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 18 ==== osd_map(64..67 src has 1..67) ==== 2834+0+0 (secure 0 0 0) 0x7f3e28095a50 con 0x55dfbde37740 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(64..67 src has 1..67) 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [64,67] > 63 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 64 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 64 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 65 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 65 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 66 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:22.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 66 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:22.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:22.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 67 2024-05-04T17:47:22.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:22.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 67 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:22.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:22.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:47:22.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:21.888+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:47:22.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:22.645+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:22.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:22.645+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:22.682 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:22.653+0000 12a65640 -1 osd.4 65 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:22.816 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:22.776+0000 ea5d640 -1 osd.7 67 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:23.011 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:22.878+0000 ea5d640 -1 osd.2 65 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:23.811 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:23.802+0000 ea5d640 -1 osd.7 68 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:23.854 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:23.820+0000 12a65640 -1 osd.4 66 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:23.869 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:23.798+0000 12a65640 -1 osd.6 64 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:23.930 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:23.916+0000 ea5d640 -1 osd.2 65 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:24.831 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:24.822+0000 ea5d640 -1 osd.7 69 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:25.207 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:47:25.185+0000 12a65640 -1 osd.0 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4666.0:15 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2024-05-04T17:47:25.254 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:25.112+0000 12a65640 -1 osd.4 68 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:25.320 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:25.221+0000 12a65640 -1 osd.6 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:25.629 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:25.490+0000 ea5d640 -1 osd.2 69 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:25.798 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:25.790+0000 ea5d640 -1 osd.7 69 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:26.230 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:47:26.207+0000 12a65640 -1 osd.0 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4666.0:15 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2024-05-04T17:47:26.445 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: Running main() from gmock_main.cc 2024-05-04T17:47:26.446 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-05-04T17:47:26.446 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [----------] Global test environment set-up. 2024-05-04T17:47:26.446 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-05-04T17:47:26.446 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-05-04T17:47:26.446 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ OK ] LibRadosPools.PoolList (8282 ms) 2024-05-04T17:47:26.446 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-05-04T17:47:26.446 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (6495 ms) 2024-05-04T17:47:26.446 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-05-04T17:47:26.446 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (8488 ms) 2024-05-04T17:47:26.446 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-05-04T17:47:26.446 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (5046 ms) 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (6972 ms) 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6031 ms) 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6237 ms) 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (6385 ms) 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (10877 ms) 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [----------] 9 tests from LibRadosPools (64813 ms total) 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [----------] Global test environment tear-down 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (64816 ms total) 2024-05-04T17:47:26.447 INFO:tasks.workunit.client.0.smithi019.stdout: api_pool: [ PASSED ] 9 tests. 2024-05-04T17:47:26.719 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:26.648+0000 12a65640 -1 osd.4 69 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:26.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:22.645+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:26.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:23.645+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:26.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:23.645+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:26.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:23.645+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:24.645+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:24.645+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:24.645+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:24.645+0000 7f3e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:25.645+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:25.645+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:25.645+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.645+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.645+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.645+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.876+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.876+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:47:26.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.876+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.876+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.876+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.876+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=68}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.876+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e08004150 con 0x55dfbdfb15a0 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 19 ==== osd_map(68..71 src has 1..71) ==== 2825+0+0 (secure 0 0 0) 0x7f3e28095a50 con 0x55dfbde37740 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(68..71 src has 1..71) 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [68,71] > 67 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 68 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 68 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 69 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:26.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 69 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:27.004 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:26.865+0000 12a65640 -1 osd.6 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:27.465 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:47:27.366+0000 12a65640 -1 osd.0 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4666.0:15 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2024-05-04T17:47:27.500 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:27.386+0000 ea5d640 -1 osd.7 71 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:27.826 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:27.813+0000 ea5d640 -1 osd.2 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:28.613 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:28.607+0000 12a65640 -1 osd.6 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:28.787 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:28.780+0000 ea5d640 -1 osd.7 72 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:28.867 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:28.683+0000 12a65640 -1 osd.4 71 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:29.011 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:28.946+0000 ea5d640 -1 osd.2 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:29.073 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:47:28.940+0000 12a65640 -1 osd.0 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4666.0:15 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2024-05-04T17:47:29.757 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:29.752+0000 ea5d640 -1 osd.7 73 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:29.835 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:29.795+0000 12a65640 -1 osd.6 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:30.146 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:30.114+0000 12a65640 -1 osd.4 72 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:30.315 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:30.256+0000 ea5d640 -1 osd.2 71 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:30.815 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:30.807+0000 ea5d640 -1 osd.7 73 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:31.187 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:31.180+0000 12a65640 -1 osd.6 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:31.225 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:31.210+0000 12a65640 -1 osd.4 73 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:31.317 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:31.310+0000 ea5d640 -1 osd.2 71 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:31.334 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:47:31.182+0000 12a65640 -1 osd.0 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4666.0:15 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2024-05-04T17:47:31.783 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:31.778+0000 ea5d640 -1 osd.7 73 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:31.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:31.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 70 2024-05-04T17:47:31.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:31.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 70 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:31.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:31.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 71 2024-05-04T17:47:31.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:31.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 71 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:31.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:31.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:47:31.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:26.879+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:47:31.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:27.645+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:27.645+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:27.645+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:28.645+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:28.645+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:28.645+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:29.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:29.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:29.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:30.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:30.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:30.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.877+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:47:31.879 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.877+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:47:31.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.877+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:47:31.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.877+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:47:31.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.877+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:47:31.880 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.877+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=72}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:47:32.185 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:32.180+0000 12a65640 -1 osd.4 73 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:32.211 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:47:32.188+0000 12a65640 -1 osd.0 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4666.0:15 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2024-05-04T17:47:32.349 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:32.330+0000 12a65640 -1 osd.6 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:32.414 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:32.327+0000 ea5d640 -1 osd.2 72 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:32.928 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:32.780+0000 ea5d640 -1 osd.7 75 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:33.443 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:33.357+0000 12a65640 -1 osd.4 75 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:33.486 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:33.477+0000 12a65640 -1 osd.6 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:33.790 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:33.638+0000 ea5d640 -1 osd.2 74 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:34.042 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:47:33.920+0000 12a65640 -1 osd.0 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4666.0:15 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2024-05-04T17:47:34.361 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:34.264+0000 ea5d640 -1 osd.7 76 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:34.416 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:34.409+0000 12a65640 -1 osd.4 76 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:34.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.877+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e08004370 con 0x55dfbdfb15a0 2024-05-04T17:47:34.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.880+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 20 ==== osd_map(72..75 src has 1..75) ==== 1456+0+0 (secure 0 0 0) 0x7f3e28095a50 con 0x55dfbde37740 2024-05-04T17:47:34.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.880+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(72..75 src has 1..75) 2024-05-04T17:47:34.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [72,75] > 71 2024-05-04T17:47:34.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 72 2024-05-04T17:47:34.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:34.647 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 72 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 73 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 73 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 74 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 74 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 75 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 75 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:34.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:47:34.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:31.881+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:47:34.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:32.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:34.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:32.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:34.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:32.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:34.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:33.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:34.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:33.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:34.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:33.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:34.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:34.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:34.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:34.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:34.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:34.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:34.761 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:34.751+0000 12a65640 -1 osd.6 74 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:35.075 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:34.884+0000 ea5d640 -1 osd.2 75 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:35.470 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:47:35.364+0000 12a65640 -1 osd.0 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4666.0:15 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2024-05-04T17:47:35.654 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:35.576+0000 12a65640 -1 osd.4 77 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:35.693 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:35.626+0000 ea5d640 -1 osd.7 77 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:35.956 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:35.880+0000 12a65640 -1 osd.6 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:35.981 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:35.907+0000 ea5d640 -1 osd.2 75 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:36.719 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:36.688+0000 ea5d640 -1 osd.7 78 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:36.845 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:36.818+0000 12a65640 -1 osd.4 78 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:36.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:34.646+0000 7f3e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:47:36.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:35.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:36.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:35.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:36.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:35.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:36.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:36.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.878+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.878+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.878+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.878+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.878+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.878+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=76}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.878+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e08004370 con 0x55dfbdfb15a0 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 21 ==== osd_map(76..79 src has 1..79) ==== 1800+0+0 (secure 0 0 0) 0x7f3e28095a50 con 0x55dfbde37740 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(76..79 src has 1..79) 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [76,79] > 75 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 76 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:36.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 76 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:36.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:36.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 77 2024-05-04T17:47:36.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:36.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 77 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:36.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:36.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 78 2024-05-04T17:47:36.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:36.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 78 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:36.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:36.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 79 2024-05-04T17:47:36.978 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:36.963+0000 ea5d640 -1 osd.2 75 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:37.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:36.985+0000 12a65640 -1 osd.6 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:37.475 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:47:37.138+0000 12a65640 -1 osd.0 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4666.0:15 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2024-05-04T17:47:37.778 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:37.761+0000 ea5d640 -1 osd.7 79 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:38.077 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:38.037+0000 ea5d640 -1 osd.2 76 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:38.081 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:38.032+0000 12a65640 -1 osd.6 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:38.247 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:38.238+0000 12a65640 -1 osd.4 78 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:38.416 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:47:38.242+0000 12a65640 -1 osd.0 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4666.0:15 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e39) 2024-05-04T17:47:38.794 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:38.786+0000 ea5d640 -1 osd.7 79 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:39.049 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:47:39.027+0000 12a65640 -1 osd.6 76 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4534.0:2 33.15 33.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-05-04T17:47:39.182 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:39.180+0000 ea5d640 -1 osd.2 77 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:39.365 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:39.355+0000 12a65640 -1 osd.4 80 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:39.849 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:39.840+0000 ea5d640 -1 osd.7 80 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:40.216 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:40.207+0000 ea5d640 -1 osd.2 77 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:40.394 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:40.388+0000 12a65640 -1 osd.4 82 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:41.086 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:36.017 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:41.150 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:36.080 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:41.213 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:36.102 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:41.217 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:36.106 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:41.231 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:36.162 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:41.238 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:36.169 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:41.332 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:36.263 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:41.345 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:36.235 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:41.410 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:36.341 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:47:41.496 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:41.411+0000 ea5d640 -1 osd.7 82 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:41.730 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:41.631+0000 ea5d640 -1 osd.2 79 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4618.0:1 41.6s0 41:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:41.760 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:41.727+0000 12a65640 -1 osd.4 82 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:42.000 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:42.000 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 79 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:42.000 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:42.000 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:47:42.000 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:36.882+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:47:42.000 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:37.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:42.000 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:37.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:37.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:38.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:38.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:38.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:39.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:39.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:39.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:40.647+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:40.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:40.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.647+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.879+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:47:42.001 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.879+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:47:42.002 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.879+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:47:42.002 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.879+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:47:42.002 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.879+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:47:42.002 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.879+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=80}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:47:42.002 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.879+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e08004370 con 0x55dfbdfb15a0 2024-05-04T17:47:42.002 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 22 ==== osd_map(80..82 src has 1..82) ==== 1175+0+0 (secure 0 0 0) 0x7f3e28095a50 con 0x55dfbde37740 2024-05-04T17:47:42.002 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(80..82 src has 1..82) 2024-05-04T17:47:42.002 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [80,82] > 79 2024-05-04T17:47:42.002 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 80 2024-05-04T17:47:42.002 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:42.521 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:42.501+0000 ea5d640 -1 osd.7 83 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:42.731 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:42.725+0000 12a65640 -1 osd.4 82 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:42.895 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:42.821+0000 ea5d640 -1 osd.2 79 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4397.0:2 29.1 29:85412baa:test-rados-api-smithi019-38807-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:43.561 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:43.548+0000 ea5d640 -1 osd.7 83 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:43.850 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:47:43.834+0000 12a65640 -1 osd.4 83 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4431.0:2 17.1a 17:5c6776db:::0.obj:head [write 0~58 in=58b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:43.996 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:47:43.983+0000 ea5d640 -1 osd.2 80 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4397.0:2 29.1 29:85412baa:test-rados-api-smithi019-38807-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29) 2024-05-04T17:47:44.666 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:44.629+0000 ea5d640 -1 osd.7 84 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 20 client.4386.d 0x55a137027bf0 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137028680 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137026df0 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a1370291e0 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137029c70 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a1370283d0 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13702a780 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13702b1f0 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137029a10 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13702c0c0 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13702afa0 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13702cfe0 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13702be40 2024-05-04T17:47:44.928 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13702de90 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13702cd10 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13702e9a0 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13702f440 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a13702dc90 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137030330 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137030a00 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137251580 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a1371a40b0 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137051700 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 0x55a137254ae0 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: started 75 aios 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137113bf0 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a136eca660 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137116940 2024-05-04T17:47:44.929 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a136ece700 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a136f80a00 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13724c200 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13724cc20 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13724d690 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13724e0e0 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13724eb30 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13724f580 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137250010 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137250b50 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137251690 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137051230 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137051cc0 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137052800 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137053340 2024-05-04T17:47:44.930 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137053e80 2024-05-04T17:47:44.931 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a1370549c0 2024-05-04T17:47:44.931 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137252150 2024-05-04T17:47:44.931 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137252bc0 2024-05-04T17:47:44.931 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137253700 2024-05-04T17:47:44.931 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137254240 2024-05-04T17:47:44.931 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137254d80 2024-05-04T17:47:44.931 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137251d80 2024-05-04T17:47:44.931 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137251ef0 2024-05-04T17:47:44.931 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137050e60 2024-05-04T17:47:44.931 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a1370510c0 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137253f70 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137253310 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137253590 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137252920 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a1370545d0 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137054890 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137053c00 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137052ff0 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137052410 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137052610 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137051980 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a1372512a0 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137251470 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a1372549d0 2024-05-04T17:47:44.932 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13724fc20 2024-05-04T17:47:44.933 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13724fe80 2024-05-04T17:47:44.933 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13724f2a0 2024-05-04T17:47:44.933 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13724e790 2024-05-04T17:47:44.933 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13724ea20 2024-05-04T17:47:44.933 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13724de80 2024-05-04T17:47:44.933 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137027060 2024-05-04T17:47:44.933 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a1372508b0 2024-05-04T17:47:44.933 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137027bf0 2024-05-04T17:47:44.933 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137028680 2024-05-04T17:47:44.933 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137026df0 2024-05-04T17:47:44.933 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a1370291e0 2024-05-04T17:47:44.933 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137029c70 2024-05-04T17:47:44.934 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a1370283d0 2024-05-04T17:47:45.687 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:45.656+0000 ea5d640 -1 osd.7 84 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:46.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting objecter _calc_target epoch 80 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:46.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:46.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 81 2024-05-04T17:47:46.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:46.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 81 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:46.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:46.881 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 82 2024-05-04T17:47:46.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:46.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 82 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:46.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:46.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:47:46.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:41.999+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:47:46.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:42.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:46.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:42.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:46.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:42.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:46.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:43.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:46.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:43.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:46.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:43.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:46.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:44.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:46.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:44.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:46.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:44.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:46.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:44.646+0000 7f3e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:47:46.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:45.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:46.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:45.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:46.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:45.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:46.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:46.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:46.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:46.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.879+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:47:46.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.879+0000 7f3e39aed640 2 client.4386.objecter tid 2 on osd.7 is laggy 2024-05-04T17:47:46.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.879+0000 7f3e39aed640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T17:47:46.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.879+0000 7f3e39aed640 10 monclient: _renew_subs 2024-05-04T17:47:46.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.879+0000 7f3e39aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T17:47:46.898 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T17:47:46.839+0000 ea5d640 -1 osd.7 86 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4386.0:2 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) 2024-05-04T17:47:47.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.879+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=83}) -- 0x7f3e080039c0 con 0x55dfbde37740 2024-05-04T17:47:47.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.879+0000 7f3e39aed640 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- ping magic: 0 -- 0x7f3e08004370 con 0x55dfbdfb15a0 2024-05-04T17:47:47.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== mon.0 v2:172.21.15.19:3300/0 23 ==== osd_map(83..87 src has 1..87) ==== 2775+0+0 (secure 0 0 0) 0x7f3e28095a50 con 0x55dfbde37740 2024-05-04T17:47:47.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(83..87 src has 1..87) 2024-05-04T17:47:47.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [83,87] > 82 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 83 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 83 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 84 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 84 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 85 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 85 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:47.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 86 2024-05-04T17:47:47.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:47.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 86 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:47.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:47.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 87 2024-05-04T17:47:47.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 10 client.4386.objecter checking op 2 2024-05-04T17:47:47.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 87 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:47.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:47.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:47:47.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:46.998+0000 7f3e35084640 20 client.4386.objecter 2 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:47:47.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:47.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:47.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:47.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:48.391 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:47.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:48.391 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 2 ==== osd_op_reply(2 exist [writefull 0~5] v43'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:47:48.391 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(2 exist [writefull 0~5] v43'1 uv1 ondisk = 0) 2024-05-04T17:47:48.391 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 2 ondisk uv 1 in 14.1 attempt 0 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 2 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e3808a640 15 client.4386.objecter _finish_op 2 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 2 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 10 librados: Objecter returned from writefull r=0 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfaec20 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 87 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:47:48.392 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:47:48.393 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [read 0~256] tid 3 osd.7 2024-05-04T17:47:48.393 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:47:48.393 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 3 2024-05-04T17:47:48.393 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 15 client.4386.objecter _send_op 3 to 14.1 on osd.7 2024-05-04T17:47:48.393 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:3 14.1 14:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87) -- 0x55dfbde47d10 con 0x55dfbdfb15a0 2024-05-04T17:47:48.393 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:48.393 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 5 client.4386.objecter 1 in flight 2024-05-04T17:47:48.393 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbde480b0 2024-05-04T17:47:48.393 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 87 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:47:48.393 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-05-04T17:47:48.393 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.a7bbaa1 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:47:49.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:49.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid noexist '@14' '@14' [read 0~256] tid 4 osd.7 2024-05-04T17:47:49.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:49.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 4 2024-05-04T17:47:49.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 15 client.4386.objecter _send_op 4 to 14.1 on osd.7 2024-05-04T17:47:49.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:4 14.1 14:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87) -- 0x55dfbde48560 con 0x55dfbdfb15a0 2024-05-04T17:47:49.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 5 2024-05-04T17:47:49.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.389+0000 7f3e39d86780 5 client.4386.objecter 2 in flight 2024-05-04T17:47:49.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:49.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:48.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 3 ==== osd_map(88..88 src has 1..88) ==== 1346+0+0 (crc 0 0 0) 0x7f3e2c00e470 con 0x55dfbdfb15a0 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(88..88 src has 1..88) 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [88,88] > 87 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 88 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 10 client.4386.objecter checking op 3 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 88 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 10 client.4386.objecter checking op 4 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 88 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:47:49.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 20 client.4386.objecter 3 14.d3a59201 osd.7 exist [read 0~256] 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.712+0000 7f3e35084640 20 client.4386.objecter 4 14.a7bbaa1 osd.7 noexist [read 0~256] 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 4 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 3 ondisk uv 1 in 14.1 attempt 0 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 5 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 3 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 15 client.4386.objecter _finish_op 3 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 5 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 3 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 5 client.4386.objecter 1 in flight 2024-05-04T17:47:49.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 5 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 4 ondisk uv 0 in 14.1 attempt 0 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 10 client.4386.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1341 ms) 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 4 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 15 client.4386.objecter _finish_op 4 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 4 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbde480b0 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 88 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:47:49.735 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:49.736 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:47:49.736 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:47:49.736 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [read 0~256] tid 5 osd.7 2024-05-04T17:47:49.736 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:47:49.736 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 5 2024-05-04T17:47:49.736 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 15 client.4386.objecter _send_op 5 to 14.1 on osd.7 2024-05-04T17:47:49.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:5 14.1 14:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88) -- 0x55dfbde48930 con 0x55dfbdfb15a0 2024-05-04T17:47:49.882 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:49.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 5 client.4386.objecter 1 in flight 2024-05-04T17:47:49.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfaec20 2024-05-04T17:47:49.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 88 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:47:49.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-05-04T17:47:49.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.a7bbaa1 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:47:49.883 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:49.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid noexist '@14' '@14' [read 0~256] tid 6 osd.7 2024-05-04T17:47:49.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:49.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 6 2024-05-04T17:47:49.884 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 15 client.4386.objecter _send_op 6 to 14.1 on osd.7 2024-05-04T17:47:49.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:6 14.1 14:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88) -- 0x55dfbdfcba50 con 0x55dfbdfb15a0 2024-05-04T17:47:49.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 5 2024-05-04T17:47:49.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.730+0000 7f3e39d86780 5 client.4386.objecter 2 in flight 2024-05-04T17:47:49.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 6 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:47:49.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-05-04T17:47:49.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:47:49.885 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 5 ondisk uv 1 in 14.1 attempt 0 2024-05-04T17:47:49.886 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 5 2024-05-04T17:47:49.886 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 5 2024-05-04T17:47:49.886 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 15 client.4386.objecter _finish_op 5 2024-05-04T17:47:49.886 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 5 2024-05-04T17:47:49.886 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 5 2024-05-04T17:47:49.886 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 5 client.4386.objecter 1 in flight 2024-05-04T17:47:49.886 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 7 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:47:49.887 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-05-04T17:47:49.887 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:47:49.887 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 6 ondisk uv 0 in 14.1 attempt 0 2024-05-04T17:47:49.887 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 10 client.4386.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-04T17:47:49.887 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:47:49.887 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 6 2024-05-04T17:47:49.888 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 15 client.4386.objecter _finish_op 6 2024-05-04T17:47:49.888 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:49.888 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 6 2024-05-04T17:47:49.888 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:47:49.888 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (150 ms) 2024-05-04T17:47:49.888 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-05-04T17:47:49.889 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfaec20 2024-05-04T17:47:49.889 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 88 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:47:49.889 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:49.889 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:47:49.889 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:47:49.889 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [read 0~256] tid 7 osd.7 2024-05-04T17:47:49.890 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:47:49.890 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 7 2024-05-04T17:47:49.890 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 15 client.4386.objecter _send_op 7 to 14.1 on osd.7 2024-05-04T17:47:49.890 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:7 14.1 14:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88) -- 0x55dfbde480b0 con 0x55dfbdfb15a0 2024-05-04T17:47:49.890 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:49.890 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 5 client.4386.objecter 1 in flight 2024-05-04T17:47:49.891 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbe00c590 2024-05-04T17:47:49.891 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 88 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:47:49.891 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-05-04T17:47:49.891 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.a7bbaa1 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:47:49.891 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:49.911 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid noexist '@14' '@14' [read 0~256] tid 8 osd.7 2024-05-04T17:47:49.912 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:49.912 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 8 2024-05-04T17:47:49.912 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 15 client.4386.objecter _send_op 8 to 14.1 on osd.7 2024-05-04T17:47:49.912 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:8 14.1 14:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88) -- 0x55dfbe00ca40 con 0x55dfbdfb15a0 2024-05-04T17:47:49.912 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 5 2024-05-04T17:47:49.913 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.880+0000 7f3e39d86780 5 client.4386.objecter 2 in flight 2024-05-04T17:47:49.913 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (29 ms) 2024-05-04T17:47:49.913 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-05-04T17:47:49.913 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 8 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:47:49.913 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-05-04T17:47:49.914 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:47:49.914 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 7 ondisk uv 1 in 14.1 attempt 0 2024-05-04T17:47:49.914 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 5 2024-05-04T17:47:49.914 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 7 2024-05-04T17:47:49.914 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 15 client.4386.objecter _finish_op 7 2024-05-04T17:47:49.915 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 5 2024-05-04T17:47:49.915 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 7 2024-05-04T17:47:49.915 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 5 client.4386.objecter 1 in flight 2024-05-04T17:47:49.915 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 9 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:47:49.915 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-05-04T17:47:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:47:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 8 ondisk uv 0 in 14.1 attempt 0 2024-05-04T17:47:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 10 client.4386.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-04T17:47:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:47:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 8 2024-05-04T17:47:49.916 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 15 client.4386.objecter _finish_op 8 2024-05-04T17:47:49.917 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:49.917 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 8 2024-05-04T17:47:54.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:47:54.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-05-04T17:47:54.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 20 librados: queue_aio_write 0x55dfbddad080 completion 0x55dfbdee7940 write_seq 1 2024-05-04T17:47:54.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbe00c590 2024-05-04T17:47:54.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 88 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:54.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:54.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:47:54.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:47:54.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [write 0~5 in=5b] tid 9 osd.7 2024-05-04T17:47:54.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 9 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 15 client.4386.objecter _send_op 9 to 14.1 on osd.7 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:9 14.1 14:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x55dfbdfaec20 con 0x55dfbdfb15a0 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 5 client.4386.objecter 1 in flight 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:49.909+0000 7f3e39d86780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:50.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:50.646+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:50.646+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:51.646+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:51.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:51.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:51.880+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:52.647+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:54.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:52.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:54.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:52.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:54.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:53.647+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:54.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:53.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:54.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:53.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:54.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:54.647+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:54.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:54.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:54.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:54.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:55.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:54.647+0000 7f3e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:47:55.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 10 ==== osd_op_reply(9 exist [write 0~5] v88'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:47:55.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(9 exist [write 0~5] v88'2 uv2 ondisk = 0) 2024-05-04T17:47:55.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:47:55.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 9 ondisk uv 2 in 14.1 attempt 0 2024-05-04T17:47:55.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:47:55.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 9 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e3808a640 15 client.4386.objecter _finish_op 9 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 9 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e3808a640 20 librados: complete_aio_write 0x55dfbdee7940 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (5192 ms) 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 20 librados: queue_aio_write 0x55dfbddad080 completion 0x55dfbdd8f020 write_seq 2 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbe00c590 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 88 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [write 0~5 in=5b] tid 10 osd.7 2024-05-04T17:47:55.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:47:55.150 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 10 2024-05-04T17:47:55.150 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 15 client.4386.objecter _send_op 10 to 14.1 on osd.7 2024-05-04T17:47:55.150 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:10 14.1 14:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x55dfbdfaec20 con 0x55dfbdfb15a0 2024-05-04T17:47:55.150 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:47:55.150 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 5 client.4386.objecter 1 in flight 2024-05-04T17:47:55.150 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.101+0000 7f3e39d86780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-05-04T17:47:58.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 11 ==== osd_map(89..92 src has 1..94) ==== 2490+0+0 (crc 0 0 0) 0x7f3e2c00e470 con 0x55dfbdfb15a0 2024-05-04T17:47:58.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(89..92 src has 1..94) 2024-05-04T17:47:58.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [89,92] > 88 2024-05-04T17:47:58.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 89 2024-05-04T17:47:58.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 10 client.4386.objecter checking op 10 2024-05-04T17:47:58.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 89 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:58.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:58.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 90 2024-05-04T17:47:58.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 10 client.4386.objecter checking op 10 2024-05-04T17:47:58.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 90 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:58.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:58.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 91 2024-05-04T17:47:58.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 10 client.4386.objecter checking op 10 2024-05-04T17:47:58.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 91 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:58.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 92 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 10 client.4386.objecter checking op 10 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 92 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.146+0000 7f3e35084640 20 client.4386.objecter 10 14.d3a59201 osd.7 exist [write 0~5 in=5b] 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.647+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:55.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:56.647+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:56.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:56.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:56.881+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:57.647+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:47:58.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:57.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:47:58.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:57.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:47:58.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:58.647+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:01.801 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:56.731 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:01.855 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:01.637+0000 ea5d640 -1 osd.3 95 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:01.893 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:56.824 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:01.979 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:56.909 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:01.983 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:56.872 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:01.984 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:56.915 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:01.985 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:56.875 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:02.055 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:03:56.986 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:02.112 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:03:57.001 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:02.171 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:03:57.102 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:03.058 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:02.969+0000 ea5d640 -1 osd.3 97 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:03.765 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:58.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:03.765 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:58.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:03.765 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:59.647+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:03.765 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:59.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:47:59.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:00.647+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:00.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:00.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:01.647+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:01.647+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:01.647+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:01.882+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:02.648+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:02.649+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:02.649+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.649+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.649+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:03.766 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.649+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 12 ==== osd_op_reply(10 exist [write 0~5] v92'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(10 exist [write 0~5] v92'3 uv3 ondisk = 0) 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 10 ondisk uv 3 in 14.1 attempt 0 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 10 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e3808a640 15 client.4386.objecter _finish_op 10 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 10 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e3808a640 20 librados: complete_aio_write 0x55dfbdd8f020 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (8662 ms) 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-05-04T17:48:03.767 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 20 librados: queue_aio_write 0x55dfbddad080 completion 0x55dfbe00c590 write_seq 3 2024-05-04T17:48:03.768 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfef580 2024-05-04T17:48:03.768 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 92 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:03.778 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:03.778 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:48:03.778 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:03.778 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [write 0~5 in=5b] tid 11 osd.7 2024-05-04T17:48:03.778 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:03.778 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 11 2024-05-04T17:48:03.778 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 15 client.4386.objecter _send_op 11 to 14.1 on osd.7 2024-05-04T17:48:03.778 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:11 14.1 14:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92) -- 0x55dfbdfefa30 con 0x55dfbdfb15a0 2024-05-04T17:48:03.778 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:03.778 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 5 client.4386.objecter 1 in flight 2024-05-04T17:48:03.778 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.764+0000 7f3e39d86780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-05-04T17:48:03.779 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 13 ==== osd_map(93..100 src has 1..100) ==== 5367+0+0 (crc 0 0 0) 0x7f3e2c00e470 con 0x55dfbdfb15a0 2024-05-04T17:48:03.781 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(93..100 src has 1..100) 2024-05-04T17:48:03.781 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [93,100] > 92 2024-05-04T17:48:03.781 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 93 2024-05-04T17:48:03.781 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 10 client.4386.objecter checking op 11 2024-05-04T17:48:03.781 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 93 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:03.781 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:03.781 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 94 2024-05-04T17:48:03.781 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 10 client.4386.objecter checking op 11 2024-05-04T17:48:03.781 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 94 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:03.781 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:03.781 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 95 2024-05-04T17:48:03.782 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 10 client.4386.objecter checking op 11 2024-05-04T17:48:03.782 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 95 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:03.782 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:03.782 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 96 2024-05-04T17:48:03.782 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 10 client.4386.objecter checking op 11 2024-05-04T17:48:04.028 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:04.019+0000 ea5d640 -1 osd.3 97 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:05.152 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:05.092+0000 ea5d640 -1 osd.3 98 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:05.609 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: Running main() from gmock_main.cc 2024-05-04T17:48:05.609 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-05-04T17:48:05.609 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: [----------] Global test environment set-up. 2024-05-04T17:48:05.609 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-05-04T17:48:05.609 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-05-04T17:48:05.609 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: handle_notify cookie 94202434295680 notify_id 176093659136 notifier_gid 4409 2024-05-04T17:48:05.609 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (32075 ms) 2024-05-04T17:48:05.609 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-05-04T17:48:05.609 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: Trying... 2024-05-04T17:48:05.610 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: handle_notify cookie 94202446421328 notify_id 296352743424 notifier_gid 4798 2024-05-04T17:48:05.610 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: Waiting for 3.000000000s 2024-05-04T17:48:05.610 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: Timed out. 2024-05-04T17:48:05.610 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: Flushing... 2024-05-04T17:48:05.610 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: Flushed... 2024-05-04T17:48:05.610 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (71449 ms) 2024-05-04T17:48:05.610 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (103524 ms total) 2024-05-04T17:48:05.610 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: 2024-05-04T17:48:05.610 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: [----------] Global test environment tear-down 2024-05-04T17:48:05.610 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (103524 ms total) 2024-05-04T17:48:05.610 INFO:tasks.workunit.client.0.smithi019.stdout: watch_notify: [ PASSED ] 2 tests. 2024-05-04T17:48:06.577 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:06.567+0000 12a65640 -1 osd.0 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:06.892 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:06.779+0000 ea5d640 -1 osd.3 100 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:07.478 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:07.463+0000 12a65640 -1 osd.4 101 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e32) 2024-05-04T17:48:07.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 96 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:07.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 97 2024-05-04T17:48:07.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 10 client.4386.objecter checking op 11 2024-05-04T17:48:07.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 97 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 98 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 10 client.4386.objecter checking op 11 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 98 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 99 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 10 client.4386.objecter checking op 11 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 99 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 100 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 10 client.4386.objecter checking op 11 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 100 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:48:07.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:03.777+0000 7f3e35084640 20 client.4386.objecter 11 14.d3a59201 osd.7 exist [write 0~5 in=5b] 2024-05-04T17:48:07.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:04.649+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:07.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:04.649+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:07.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:04.649+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:07.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:04.649+0000 7f3e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T17:48:07.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:05.649+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:07.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:05.649+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:07.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:05.649+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:07.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:06.649+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:07.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:06.649+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:07.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:06.649+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:07.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:06.883+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:48:07.652 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.649+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:07.807 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.649+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:07.807 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.649+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 14 ==== osd_op_reply(11 exist [write 0~5] v100'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(11 exist [write 0~5] v100'4 uv4 ondisk = 0) 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 11 ondisk uv 4 in 14.1 attempt 0 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 11 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e3808a640 15 client.4386.objecter _finish_op 11 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 11 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e3808a640 20 librados: complete_aio_write 0x55dfbe00c590 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (4042 ms) 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfcbdf0 2024-05-04T17:48:07.808 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 100 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.809 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.809 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:48:07.809 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:07.809 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [read 0~0] tid 12 osd.7 2024-05-04T17:48:07.809 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:07.809 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 12 2024-05-04T17:48:07.809 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 15 client.4386.objecter _send_op 12 to 14.1 on osd.7 2024-05-04T17:48:07.809 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:12 14.1 14:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100) -- 0x55dfbdfcc2a0 con 0x55dfbdfb15a0 2024-05-04T17:48:07.809 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.809 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 5 client.4386.objecter 1 in flight 2024-05-04T17:48:07.809 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfcc640 2024-05-04T17:48:07.809 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 100 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.817 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-05-04T17:48:07.817 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.a7bbaa1 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:48:07.817 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.817 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid noexist '@14' '@14' [read 0~0] tid 13 osd.7 2024-05-04T17:48:07.817 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.817 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.806+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 13 2024-05-04T17:48:07.817 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.807+0000 7f3e39d86780 15 client.4386.objecter _send_op 13 to 14.1 on osd.7 2024-05-04T17:48:07.817 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.807+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:13 14.1 14:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100) -- 0x55dfbdfefe60 con 0x55dfbdfb15a0 2024-05-04T17:48:07.817 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.807+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 5 2024-05-04T17:48:07.817 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.807+0000 7f3e39d86780 5 client.4386.objecter 2 in flight 2024-05-04T17:48:07.817 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 15 ==== osd_map(101..104 src has 1..104) ==== 1813+0+0 (crc 0 0 0) 0x7f3e2c00e470 con 0x55dfbdfb15a0 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(101..104 src has 1..104) 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [101,104] > 100 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 101 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 10 client.4386.objecter checking op 12 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 101 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 10 client.4386.objecter checking op 13 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 101 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 102 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 10 client.4386.objecter checking op 12 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 102 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.818 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 10 client.4386.objecter checking op 13 2024-05-04T17:48:07.819 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 102 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.819 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-05-04T17:48:07.819 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 103 2024-05-04T17:48:07.822 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 10 client.4386.objecter checking op 12 2024-05-04T17:48:07.822 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 103 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.822 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.822 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 10 client.4386.objecter checking op 13 2024-05-04T17:48:07.822 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 103 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.822 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-05-04T17:48:07.822 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 104 2024-05-04T17:48:07.822 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 10 client.4386.objecter checking op 12 2024-05-04T17:48:07.822 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 104 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.822 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.822 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 10 client.4386.objecter checking op 13 2024-05-04T17:48:07.822 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 104 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter 12 14.d3a59201 osd.7 exist [read 0~0] 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.816+0000 7f3e35084640 20 client.4386.objecter 13 14.a7bbaa1 osd.7 noexist [read 0~0] 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.818+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 16 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.818+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.818+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.818+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 12 ondisk uv 4 in 14.1 attempt 0 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.818+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 5 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.818+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 12 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.818+0000 7f3e3808a640 15 client.4386.objecter _finish_op 12 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.818+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 5 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.818+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 12 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.818+0000 7f3e3808a640 5 client.4386.objecter 1 in flight 2024-05-04T17:48:07.823 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 17 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 13 ondisk uv 0 in 14.1 attempt 0 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e3808a640 10 client.4386.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 13 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e3808a640 15 client.4386.objecter _finish_op 13 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 13 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (15 ms) 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfcbf00 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 104 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.824 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [read 0~0] tid 14 osd.7 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 14 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 15 client.4386.objecter _send_op 14 to 14.1 on osd.7 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:14 14.1 14:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e104) -- 0x55dfbdfcc3b0 con 0x55dfbdfb15a0 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 5 client.4386.objecter 1 in flight 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfefe60 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 104 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.a7bbaa1 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.825 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid noexist '@14' '@14' [read 0~0] tid 15 osd.7 2024-05-04T17:48:07.826 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 15 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 15 client.4386.objecter _send_op 15 to 14.1 on osd.7 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:15 14.1 14:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e104) -- 0x55dfbdfcc990 con 0x55dfbdfb15a0 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 5 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.821+0000 7f3e39d86780 5 client.4386.objecter 2 in flight 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 18 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 14 ondisk uv 4 in 14.1 attempt 0 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 5 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 14 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 15 client.4386.objecter _finish_op 14 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 5 2024-05-04T17:48:07.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 14 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 5 client.4386.objecter 1 in flight 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 19 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 15 ondisk uv 0 in 14.1 attempt 0 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 10 client.4386.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 15 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 15 client.4386.objecter _finish_op 15 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 15 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (11 ms) 2024-05-04T17:48:07.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-05-04T17:48:07.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfeef10 2024-05-04T17:48:07.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 104 base exist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:48:07.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:07.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [read 0~0] tid 16 osd.7 2024-05-04T17:48:07.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:07.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 16 2024-05-04T17:48:07.847 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 15 client.4386.objecter _send_op 16 to 14.1 on osd.7 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:16 14.1 14:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e104) -- 0x55dfbdfcc990 con 0x55dfbdfb15a0 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 5 client.4386.objecter 1 in flight 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfef3c0 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 104 base noexist @14 precalc_pgid 0 pgid 0.0 is_read 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 20 client.4386.objecter _calc_target target noexist @14 -> pgid 14.a7bbaa1 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.a7bbaa1 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid noexist '@14' '@14' [read 0~0] tid 17 osd.7 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 17 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 15 client.4386.objecter _send_op 17 to 14.1 on osd.7 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:17 14.1 14:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e104) -- 0x55dfbdfef870 con 0x55dfbdfb15a0 2024-05-04T17:48:07.848 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 5 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.832+0000 7f3e39d86780 5 client.4386.objecter 2 in flight 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 20 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 16 ondisk uv 4 in 14.1 attempt 0 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 5 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 16 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 15 client.4386.objecter _finish_op 16 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 5 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 16 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 5 client.4386.objecter 1 in flight 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 21 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:48:07.849 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 17 ondisk uv 0 in 14.1 attempt 0 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 10 client.4386.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 17 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 15 client.4386.objecter _finish_op 17 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 17 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (14 ms) 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 20 librados: queue_aio_write 0x55dfbddad080 completion 0x55dfbdfcc860 write_seq 4 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfcbf00 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 104 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:48:07.850 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:07.851 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [writefull 0~5 in=5b] tid 18 osd.7 2024-05-04T17:48:07.851 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:07.851 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 18 2024-05-04T17:48:07.898 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:07.816+0000 12a65640 -1 osd.0 102 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:07.994 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:07.934+0000 ea5d640 -1 osd.3 101 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:09.019 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:08.943+0000 12a65640 -1 osd.4 102 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e32) 2024-05-04T17:48:09.566 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:09.499+0000 12a65640 -1 osd.0 103 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:09.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 15 client.4386.objecter _send_op 18 to 14.1 on osd.7 2024-05-04T17:48:09.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:18 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x55dfbdfcc970 con 0x55dfbdfb15a0 2024-05-04T17:48:09.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:09.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:07.846+0000 7f3e39d86780 5 client.4386.objecter 1 in flight 2024-05-04T17:48:09.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:08.649+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:09.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:08.649+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:09.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:08.649+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:09.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.649+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:09.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.649+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:09.833 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.649+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 22 ==== osd_op_reply(18 exist [writefull 0~5] v104'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(18 exist [writefull 0~5] v104'5 uv5 ondisk = 0) 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 18 ondisk uv 5 in 14.1 attempt 0 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 18 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e3808a640 15 client.4386.objecter _finish_op 18 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 18 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e3808a640 20 librados: complete_aio_write 0x55dfbdfcc860 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1986 ms) 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-05-04T17:48:09.834 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 20 librados: queue_aio_write 0x55dfbddad080 completion 0x55dfbdfcbdf0 write_seq 5 2024-05-04T17:48:09.835 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfcc860 2024-05-04T17:48:09.835 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 104 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:09.835 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:09.835 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:48:09.835 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:09.835 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [writefull 0~5 in=5b] tid 19 osd.7 2024-05-04T17:48:09.872 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:09.831+0000 ea5d640 -1 osd.3 102 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:10.425 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:10.344+0000 12a65640 -1 osd.4 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4424.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e32) 2024-05-04T17:48:10.551 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:10.539+0000 12a65640 -1 osd.0 103 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:11.109 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:11.099+0000 ea5d640 -1 osd.3 103 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:11.481 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:11.481 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 19 2024-05-04T17:48:11.481 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 15 client.4386.objecter _send_op 19 to 14.1 on osd.7 2024-05-04T17:48:11.481 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:19 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x55dfbdfcbf00 con 0x55dfbdfb15a0 2024-05-04T17:48:11.481 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:11.481 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:09.832+0000 7f3e39d86780 5 client.4386.objecter 1 in flight 2024-05-04T17:48:11.481 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:10.649+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:11.481 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:10.649+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:11.481 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:10.649+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:11.481 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 23 ==== osd_op_reply(19 exist [writefull 0~5] v104'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(19 exist [writefull 0~5] v104'6 uv6 ondisk = 0) 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 19 ondisk uv 6 in 14.1 attempt 0 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 19 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e3808a640 15 client.4386.objecter _finish_op 19 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 19 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e3808a640 20 librados: complete_aio_write 0x55dfbdfcbdf0 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1648 ms) 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 20 librados: queue_aio_write 0x55dfbddad080 completion 0x55dfbdfcc530 write_seq 6 2024-05-04T17:48:11.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 10 client.4386.objecter _op_submit op 0x55dfbdfcc860 2024-05-04T17:48:11.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 20 client.4386.objecter _calc_target epoch 104 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:11.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:11.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 10 client.4386.objecter _calc_target raw pgid 14.d3a59201 -> actual 14.1 acting [7,6] primary 7 2024-05-04T17:48:11.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 20 client.4386.objecter _get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:11.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 10 client.4386.objecter _op_submit oid exist '@14' '@14' [writefull 0~5 in=5b] tid 20 osd.7 2024-05-04T17:48:11.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 20 client.4386.objecter get_session s=0x55dfbdfb1080 osd=7 3 2024-05-04T17:48:12.314 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:12.257+0000 12a65640 -1 osd.0 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:12.443 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:12.360+0000 ea5d640 -1 osd.3 104 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:13.389 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:13.299+0000 12a65640 -1 osd.4 105 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:13.466 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:13.447+0000 ea5d640 -1 osd.3 105 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 15 client.4386.objecter _session_op_assign 7 20 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 15 client.4386.objecter _send_op 20 to 14.1 on osd.7 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- osd_op(unknown.0.0:20 14.1 14:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104) -- 0x55dfbdfcbf00 con 0x55dfbdfb15a0 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.480+0000 7f3e39d86780 5 client.4386.objecter 1 in flight 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.498+0000 7f3e35084640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 24 ==== osd_map(105..105 src has 1..105) ==== 1054+0+0 (crc 0 0 0) 0x7f3e2c00eba0 con 0x55dfbdfb15a0 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.498+0000 7f3e35084640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_map(105..105 src has 1..105) 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.498+0000 7f3e35084640 3 client.4386.objecter handle_osd_map got epochs [105,105] > 104 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.498+0000 7f3e35084640 3 client.4386.objecter handle_osd_map decoding incremental epoch 105 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.498+0000 7f3e35084640 10 client.4386.objecter checking op 20 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.498+0000 7f3e35084640 20 client.4386.objecter _calc_target epoch 105 base exist @14 precalc_pgid 0 pgid 0.0 is_write 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.498+0000 7f3e35084640 20 client.4386.objecter _calc_target target exist @14 -> pgid 14.d3a59201 2024-05-04T17:48:13.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.498+0000 7f3e35084640 20 client.4386.objecter dump_active .. 0 homeless 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.498+0000 7f3e35084640 20 client.4386.objecter 20 14.d3a59201 osd.7 exist [writefull 0~5 in=5b] 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.649+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.649+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.649+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:11.884+0000 7f3e39aed640 10 client.4386.objecter tick 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:12.649+0000 7f3e1ffff640 10 monclient: tick 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:12.649+0000 7f3e1ffff640 10 monclient: _check_auth_tickets 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:12.649+0000 7f3e1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 <== osd.7 v2:172.21.15.186:6826/1403900459 25 ==== osd_op_reply(20 exist [writefull 0~5] v105'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f3e2c00e870 con 0x55dfbdfb15a0 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e3808a640 10 client.4386.objecter ms_dispatch 0x55dfbdde13a0 osd_op_reply(20 exist [writefull 0~5] v105'7 uv7 ondisk = 0) 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e3808a640 10 client.4386.objecter in handle_osd_op_reply 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e3808a640 7 client.4386.objecter handle_osd_op_reply 20 ondisk uv 7 in 14.1 attempt 0 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e3808a640 10 client.4386.objecter op 0 rval 0 len 0 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e3808a640 15 client.4386.objecter handle_osd_op_reply completed tid 20 2024-05-04T17:48:13.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e3808a640 15 client.4386.objecter _finish_op 20 2024-05-04T17:48:13.553 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e3808a640 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 4 2024-05-04T17:48:13.555 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e3808a640 15 client.4386.objecter _session_op_remove 7 20 2024-05-04T17:48:13.555 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e3808a640 5 client.4386.objecter 0 in flight 2024-05-04T17:48:13.555 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e3808a640 20 librados: complete_aio_write 0x55dfbdfcc530 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2069 ms) 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 10 librados: watch_flush enter 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 10 librados: watch_flush exit 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 10 client.4386.objecter close_session for osd.7 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x55dfbdfb15a0 msgr2=0x55dfbdee7010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x55dfbdfb15a0 0x55dfbdee7010 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 20 client.4386.objecter put_session s=0x55dfbdfb1080 osd=7 1 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 20 client.4386.objecter shutdown clearing up homeless session... 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 10 client.4386.objecter successfully canceled tick 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f3e2406ba30 msgr2=0x7f3e2406def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f3e2406ba30 0x7f3e2406def0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3e30006fd0 tx=0x7f3e30008040 comp rx=0 tx=0).stop 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 10 monclient: shutdown 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T17:48:13.556 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55dfbde37740 msgr2=0x55dfbdfb06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T17:48:13.557 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 1 --2- 172.21.15.19:0/1329173258 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x55dfbde37740 0x55dfbdfb06b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3e28005cf0 tx=0x7f3e28053530 comp rx=0 tx=0).stop 2024-05-04T17:48:13.557 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e3808a640 1 -- 172.21.15.19:0/1329173258 reap_dead start 2024-05-04T17:48:13.557 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 shutdown_connections 2024-05-04T17:48:13.557 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 >> 172.21.15.19:0/1329173258 conn(0x55dfbde797f0 msgr2=0x55dfbdf02070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T17:48:13.557 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 shutdown_connections 2024-05-04T17:48:13.557 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 1 -- 172.21.15.19:0/1329173258 wait complete. 2024-05-04T17:48:13.557 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.550+0000 7f3e39d86780 1 librados: shutdown 2024-05-04T17:48:13.557 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [----------] 12 tests from AsioRados (25159 ms total) 2024-05-04T17:48:13.557 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: 2024-05-04T17:48:13.557 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [----------] Global test environment tear-down 2024-05-04T17:48:13.557 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (111947 ms total) 2024-05-04T17:48:13.557 INFO:tasks.workunit.client.0.smithi019.stdout: api_asio: [ PASSED ] 12 tests. 2024-05-04T17:48:13.584 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:13.556+0000 12a65640 -1 osd.0 105 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:14.665 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:14.439+0000 12a65640 -1 osd.4 107 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:14.730 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:14.591+0000 ea5d640 -1 osd.3 107 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:15.041 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:14.927+0000 12a65640 -1 osd.0 107 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:16.016 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:16.006+0000 12a65640 -1 osd.0 109 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4604.0:3 62.14 62.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e59) 2024-05-04T17:48:16.433 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:16.423+0000 12a65640 -1 osd.4 109 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:17.362 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:17.293+0000 12a65640 -1 osd.0 109 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:18.110 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:18.078+0000 12a65640 -1 osd.4 111 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:18.778 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:18.673+0000 12a65640 -1 osd.0 111 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:19.349 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:19.242+0000 12a65640 -1 osd.4 112 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:19.703 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:19.697+0000 12a65640 -1 osd.0 111 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:20.685 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:20.674+0000 12a65640 -1 osd.4 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:21.149 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:21.035+0000 12a65640 -1 osd.0 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:22.078 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:22.065+0000 12a65640 -1 osd.0 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:22.307 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:22.171+0000 12a65640 -1 osd.4 114 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:23.632 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:23.591+0000 12a65640 -1 osd.4 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:23.839 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:23.798+0000 12a65640 -1 osd.0 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:24.772 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:24.757+0000 12a65640 -1 osd.4 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:26.034 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:26.002+0000 12a65640 -1 osd.4 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:26.256 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:26.138+0000 12a65640 -1 osd.0 117 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:27.492 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:27.475+0000 12a65640 -1 osd.0 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:27.910 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:27.874+0000 ea5d640 -1 osd.2 114 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:28.540 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:28.534+0000 12a65640 -1 osd.0 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:29.185 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:29.072+0000 ea5d640 -1 osd.2 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:30.204 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:30.026+0000 12a65640 -1 osd.0 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:30.477 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:30.382+0000 ea5d640 -1 osd.2 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:31.403 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:31.396+0000 12a65640 -1 osd.0 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:31.558 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:31.495+0000 ea5d640 -1 osd.2 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:32.629 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:32.544+0000 12a65640 -1 osd.0 122 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:32.734 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:32.723+0000 12a65640 -1 osd.4 123 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:33.148 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:33.136+0000 ea5d640 -1 osd.2 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:33.768 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:33.763+0000 12a65640 -1 osd.0 124 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:33.989 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:33.842+0000 12a65640 -1 osd.4 125 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:34.226 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:34.203+0000 ea5d640 -1 osd.2 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:34.826 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:34.811+0000 12a65640 -1 osd.0 124 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:35.355 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:35.283+0000 ea5d640 -1 osd.2 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:35.363 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:35.350+0000 12a65640 -1 osd.4 127 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:36.296 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:36.291+0000 12a65640 -1 osd.0 126 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:36.517 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:36.402+0000 ea5d640 -1 osd.2 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:36.958 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:36.948+0000 12a65640 -1 osd.4 128 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:37.608 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:37.461+0000 ea5d640 -1 osd.2 122 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:38.127 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:38.117+0000 12a65640 -1 osd.4 129 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:38.169 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:38.073+0000 12a65640 -1 osd.0 128 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:38.763 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:38.752+0000 ea5d640 -1 osd.2 123 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:39.297 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:39.284+0000 12a65640 -1 osd.0 129 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:39.308 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:39.291+0000 12a65640 -1 osd.4 130 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:39.961 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:39.914+0000 ea5d640 -1 osd.2 124 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:40.607 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:40.576+0000 12a65640 -1 osd.0 130 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:40.913 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:40.796+0000 12a65640 -1 osd.4 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:41.520 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:41.416+0000 ea5d640 -1 osd.2 126 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:41.727 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:41.718+0000 12a65640 -1 osd.0 131 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:42.050 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:41.959+0000 12a65640 -1 osd.4 133 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:43.260 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:43.211+0000 ea5d640 -1 osd.2 127 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:43.420 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:43.298+0000 12a65640 -1 osd.4 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:43.426 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:43.312+0000 12a65640 -1 osd.0 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:44.337 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:44.332+0000 12a65640 -1 osd.0 133 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:44.391 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:44.319+0000 ea5d640 -1 osd.2 128 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:45.370 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:45.351+0000 12a65640 -1 osd.0 133 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:45.398 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:45.353+0000 ea5d640 -1 osd.2 128 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:46.349 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:46.344+0000 ea5d640 -1 osd.2 130 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:46.458 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:46.409+0000 12a65640 -1 osd.0 134 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:46.491 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:46.402+0000 ea5d640 -1 osd.3 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:48:46.748 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:46.694+0000 12a65640 -1 osd.4 138 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:47.579 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:47.538+0000 12a65640 -1 osd.0 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:47.955 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:47.786+0000 ea5d640 -1 osd.3 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:48:48.919 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:48.865+0000 12a65640 -1 osd.4 139 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:49.009 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:48.985+0000 12a65640 -1 osd.0 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:49.504 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:49.390+0000 ea5d640 -1 osd.2 133 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:49.634 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:49.594+0000 ea5d640 -1 osd.3 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:48:49.999 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:49.981+0000 12a65640 -1 osd.4 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:50.262 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:50.089+0000 12a65640 -1 osd.0 137 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:51.090 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:50.949+0000 ea5d640 -1 osd.3 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:48:51.336 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:51.181+0000 12a65640 -1 osd.0 138 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:51.417 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T17:48:51.366+0000 12a65640 -1 osd.4 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4852.0:2 76.0 76:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82) 2024-05-04T17:48:51.440 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:51.375+0000 ea5d640 -1 osd.2 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:52.343 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:52.252+0000 12a65640 -1 osd.0 139 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:52.406 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:04:47.336 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:52.410 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:04:47.341 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:52.478 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:52.469+0000 ea5d640 -1 osd.2 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:52.481 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:04:47.370 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:52.486 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:04:47.375 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:52.492 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:04:47.423 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:52.501 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:04:47.432 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:52.761 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:04:47.650 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:52.781 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:04:47.712 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:52.828 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:04:47.759 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:48:53.028 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:53.005+0000 ea5d640 -1 osd.3 140 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:53.319 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:53.297+0000 12a65640 -1 osd.0 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:53.983 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:53.794+0000 ea5d640 -1 osd.2 138 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:54.185 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:54.145+0000 ea5d640 -1 osd.3 141 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:54.339 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:54.325+0000 12a65640 -1 osd.0 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:55.488 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:55.403+0000 ea5d640 -1 osd.3 141 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:55.759 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:55.750+0000 12a65640 -1 osd.0 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:55.978 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:48:55.962+0000 ea5d640 -1 osd.2 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:48:56.888 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:56.775+0000 ea5d640 -1 osd.3 143 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:57.094 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:57.045+0000 12a65640 -1 osd.0 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:58.005 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:57.979+0000 ea5d640 -1 osd.3 143 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:48:58.308 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:48:58.130+0000 12a65640 -1 osd.0 144 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:48:59.290 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:48:59.141+0000 ea5d640 -1 osd.3 144 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:49:00.264 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:00.249+0000 12a65640 -1 osd.0 146 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:00.580 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:00.567+0000 ea5d640 -1 osd.3 145 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:49:01.429 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:01.420+0000 12a65640 -1 osd.0 146 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:01.778 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:01.678+0000 ea5d640 -1 osd.3 146 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:02.084 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:02.011+0000 ea5d640 -1 osd.2 145 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:02.928 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:02.872+0000 12a65640 -1 osd.0 147 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:03.254 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:03.182+0000 ea5d640 -1 osd.2 146 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:03.276 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:03.201+0000 ea5d640 -1 osd.3 147 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:04.200 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:04.103+0000 12a65640 -1 osd.0 148 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:04.288 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:04.246+0000 ea5d640 -1 osd.3 148 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:04.451 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:04.359+0000 ea5d640 -1 osd.2 148 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:05.249 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:05.243+0000 ea5d640 -1 osd.3 148 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:05.439 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:05.195+0000 12a65640 -1 osd.0 149 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:05.508 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:05.486+0000 ea5d640 -1 osd.2 148 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:06.361 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:06.322+0000 12a65640 -1 osd.0 150 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:06.482 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:06.467+0000 ea5d640 -1 osd.3 149 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:06.940 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:06.719+0000 ea5d640 -1 osd.2 150 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:08.633 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:08.506+0000 ea5d640 -1 osd.2 151 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:08.958 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:08.732+0000 ea5d640 -1 osd.3 151 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:09.686 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:09.614+0000 12a65640 -1 osd.0 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:10.105 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:10.051+0000 ea5d640 -1 osd.2 152 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:10.128 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:10.051+0000 ea5d640 -1 osd.3 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:10.829 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:10.698+0000 12a65640 -1 osd.0 155 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:11.307 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:11.207+0000 ea5d640 -1 osd.2 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:11.391 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:11.371+0000 ea5d640 -1 osd.3 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:12.409 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:12.226+0000 12a65640 -1 osd.0 157 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:12.528 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:12.486+0000 ea5d640 -1 osd.3 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:12.887 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:07.817 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:12.890 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:07.821 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:12.911 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:07.800 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:12.915 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:07.804 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:12.974 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:07.905 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:12.976 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:07.906 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:13.053 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:07.942 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:13.106 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:08.036 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:13.245 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:08.175 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:13.438 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:13.363+0000 12a65640 -1 osd.0 158 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:13.847 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:13.779+0000 ea5d640 -1 osd.3 157 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:14.014 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:13.833+0000 ea5d640 -1 osd.2 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:14.733 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:14.525+0000 12a65640 -1 osd.0 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:14.844 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:14.838+0000 ea5d640 -1 osd.3 157 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:15.494 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:15.489+0000 ea5d640 -1 osd.2 156 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:15.823 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:15.704+0000 12a65640 -1 osd.0 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:16.106 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:16.080+0000 ea5d640 -1 osd.3 158 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:16.924 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:16.779+0000 ea5d640 -1 osd.2 157 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:17.074 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:16.850+0000 12a65640 -1 osd.0 161 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:17.471 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:17.307+0000 ea5d640 -1 osd.3 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:17.887 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:17.840+0000 ea5d640 -1 osd.2 158 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:17.951 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:17.937+0000 12a65640 -1 osd.0 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:18.503 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:18.495+0000 ea5d640 -1 osd.3 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:19.210 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:19.139+0000 12a65640 -1 osd.0 163 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:19.292 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:19.278+0000 ea5d640 -1 osd.2 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:20.053 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:20.030+0000 ea5d640 -1 osd.3 161 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:20.471 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:20.428+0000 12a65640 -1 osd.0 164 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:20.635 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:20.581+0000 ea5d640 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:21.887 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:21.786+0000 ea5d640 -1 osd.3 163 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:21.940 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:21.903+0000 12a65640 -1 osd.0 165 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:21.968 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:21.963+0000 ea5d640 -1 osd.2 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:22.952 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:22.944+0000 12a65640 -1 osd.0 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:23.322 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:23.172+0000 ea5d640 -1 osd.3 164 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:23.370 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:23.215+0000 ea5d640 -1 osd.2 164 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:24.099 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:23.990+0000 12a65640 -1 osd.0 167 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:24.657 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:24.579+0000 ea5d640 -1 osd.2 165 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:25.210 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:24.878+0000 ea5d640 -1 osd.3 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:25.684 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:25.589+0000 12a65640 -1 osd.0 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:25.763 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:25.750+0000 ea5d640 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:26.123 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:26.018+0000 ea5d640 -1 osd.3 167 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:27.199 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:27.164+0000 ea5d640 -1 osd.3 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:27.209 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:27.189+0000 ea5d640 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:27.498 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:27.389+0000 12a65640 -1 osd.0 171 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:28.275 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:28.256+0000 ea5d640 -1 osd.2 170 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:28.633 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:28.504+0000 12a65640 -1 osd.0 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:28.760 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:28.627+0000 ea5d640 -1 osd.3 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:29.538 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:29.511+0000 ea5d640 -1 osd.2 170 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:29.717 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:29.586+0000 12a65640 -1 osd.0 173 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:30.446 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:30.314+0000 ea5d640 -1 osd.3 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4456.0:9 3.1d 3:bfd3e27c:test-rados-api-smithi019-38434-6::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109) 2024-05-04T17:49:30.603 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:30.575+0000 ea5d640 -1 osd.2 171 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:30.731 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:30.717+0000 12a65640 -1 osd.0 174 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:31.861 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:31.821+0000 ea5d640 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:32.108 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:31.906+0000 12a65640 -1 osd.0 175 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:32.768 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:32.758+0000 ea5d640 -1 osd.3 173 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:49:32.987 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:32.905+0000 ea5d640 -1 osd.2 173 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4867.0:2 78.15 78:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87) 2024-05-04T17:49:33.241 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:33.060+0000 12a65640 -1 osd.0 176 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:33.721 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:28.651 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:33.727 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:28.656 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:33.799 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:28.730 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:33.801 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:28.732 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:33.818 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:28.707 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:33.821 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:28.710 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:33.867 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:28.797 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:33.945 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:28.876 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:33.955 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:28.845 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:34.063 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:34.058+0000 12a65640 -1 osd.0 176 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:34.543 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:34.508+0000 ea5d640 -1 osd.3 175 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:49:35.494 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:35.485+0000 12a65640 -1 osd.0 178 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:35.899 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:35.783+0000 ea5d640 -1 osd.3 176 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:49:39.190 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:39.069+0000 ea5d640 -1 osd.2 179 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:39.435 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:39.425+0000 12a65640 -1 osd.0 180 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:40.876 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:40.844+0000 ea5d640 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:42.289 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:42.168+0000 ea5d640 -1 osd.2 182 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:42.408 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:42.392+0000 12a65640 -1 osd.0 181 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4376.0:8 21.17 21:e89887a2:test-rados-api-smithi019-38686-4:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:a:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2024-05-04T17:49:43.001 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:42.924+0000 ea5d640 -1 osd.3 178 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:49:43.320 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:43.205+0000 ea5d640 -1 osd.2 183 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:44.379 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:44.329+0000 ea5d640 -1 osd.2 185 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:44.424 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:44.363+0000 12a65640 -1 osd.0 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:13 62.b 62:d1337354:::obj10:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e156) 2024-05-04T17:49:44.432 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:44.349+0000 ea5d640 -1 osd.3 183 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:49:45.929 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:45.864+0000 ea5d640 -1 osd.3 183 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:49:46.492 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:46.440+0000 12a65640 -1 osd.0 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:13 62.b 62:d1337354:::obj10:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e156) 2024-05-04T17:49:46.709 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:46.693+0000 ea5d640 -1 osd.2 186 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:47.312 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:47.260+0000 ea5d640 -1 osd.3 186 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:49:47.576 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:47.552+0000 12a65640 -1 osd.0 186 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:13 62.b 62:d1337354:::obj10:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e156) 2024-05-04T17:49:48.209 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:48.029+0000 ea5d640 -1 osd.2 187 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:48.815 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:48.680+0000 ea5d640 -1 osd.3 186 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:49:48.864 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T17:49:48.811+0000 12a65640 -1 osd.0 187 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4604.0:13 62.b 62:d1337354:::obj10:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e156) 2024-05-04T17:49:49.453 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:49.362+0000 ea5d640 -1 osd.2 188 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:50.397 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:50.255+0000 ea5d640 -1 osd.3 187 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:49:50.637 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:50.556+0000 ea5d640 -1 osd.2 189 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:51.922 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:51.904+0000 ea5d640 -1 osd.3 189 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73.780569b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:49:52.202 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:52.068+0000 ea5d640 -1 osd.2 192 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:53.241 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:53.231+0000 ea5d640 -1 osd.3 190 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:49:53.276 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:53.158+0000 ea5d640 -1 osd.2 195 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:53.558 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:48.489 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:53.614 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:48.503 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:53.615 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:48.504 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:53.656 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:48.586 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:53.697 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:05:48.586 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:53.722 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:48.653 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:53.726 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:48.657 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:53.779 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:05:48.710 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:53.837 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:05:48.767 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:49:54.499 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:54.384+0000 ea5d640 -1 osd.2 196 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:54.670 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:54.488+0000 ea5d640 -1 osd.3 191 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:49:55.420 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:55.416+0000 ea5d640 -1 osd.2 196 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:55.751 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:55.595+0000 ea5d640 -1 osd.3 192 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:49:56.482 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:56.472+0000 ea5d640 -1 osd.2 197 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93.78b24956 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:56.989 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:56.973+0000 ea5d640 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:49:57.641 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:57.630+0000 ea5d640 -1 osd.2 198 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93:6a924d1e:test-rados-api-smithi019-38866-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:58.247 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:58.229+0000 ea5d640 -1 osd.3 195 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:49:58.352 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:49:58.344+0000 12a65640 -1 osd.6 198 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4724.0:2 97.6 97.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e174) 2024-05-04T17:49:58.642 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:58.636+0000 ea5d640 -1 osd.2 198 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93:6a924d1e:test-rados-api-smithi019-38866-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:59.345 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:49:59.336+0000 12a65640 -1 osd.6 198 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4724.0:2 97.6 97:602f83fe:::foo:head [append 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174) 2024-05-04T17:49:59.797 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:49:59.642+0000 ea5d640 -1 osd.2 200 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93:6a924d1e:test-rados-api-smithi019-38866-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:49:59.878 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:49:59.828+0000 ea5d640 -1 osd.3 196 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:00.631 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:50:00.624+0000 ea5d640 -1 osd.2 200 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93:6a924d1e:test-rados-api-smithi019-38866-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:50:01.033 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T17:50:01.020+0000 12a65640 -1 osd.6 200 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4724.0:2 97.6 97:602f83fe:::foo:head [append 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174) 2024-05-04T17:50:01.228 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:01.122+0000 ea5d640 -1 osd.3 198 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:02.013 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:50:01.948+0000 ea5d640 -1 osd.2 201 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4942.0:1 93.6s0 93:6a924d1e:test-rados-api-smithi019-38866-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153) 2024-05-04T17:50:03.767 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:03.671+0000 ea5d640 -1 osd.3 200 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:03.956 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:50:03.880+0000 12a65640 -1 osd.1 202 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4969.0:3 98.0 98.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e179) 2024-05-04T17:50:04.241 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T17:50:04.226+0000 ea5d640 -1 osd.2 203 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4700.0:3 90.1a 90:5bff5056:test-rados-api-smithi019-38807-7::foo:head [write 0~128 in=128b] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e150) 2024-05-04T17:50:05.068 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:50:05.050+0000 12a65640 -1 osd.1 203 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4969.0:3 98.0 98.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e179) 2024-05-04T17:50:05.102 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:05.083+0000 ea5d640 -1 osd.3 201 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:05.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-05-04T17:50:05.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-05-04T17:50:05.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-05-04T17:50:05.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-05-04T17:50:05.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (223998 ms) 2024-05-04T17:50:05.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (223998 ms total) 2024-05-04T17:50:05.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_cls_remote_reads: 2024-05-04T17:50:05.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-05-04T17:50:05.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (223998 ms total) 2024-05-04T17:50:05.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-05-04T17:50:06.145 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T17:50:06.074+0000 12a65640 -1 osd.1 204 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4969.0:3 98.0 98.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e179) 2024-05-04T17:50:06.356 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:06.287+0000 ea5d640 -1 osd.3 202 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:07.512 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:07.384+0000 ea5d640 -1 osd.3 203 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:08.775 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:08.689+0000 ea5d640 -1 osd.3 204 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:12.475 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:12.224+0000 ea5d640 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:13.174 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:12.919+0000 ea5d640 -1 osd.3 206 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:50:14.193 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:13.938+0000 ea5d640 -1 osd.3 207 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:14.372 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:14.129+0000 ea5d640 -1 osd.3 207 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:50:15.550 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:15.241+0000 ea5d640 -1 osd.3 207 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:15.654 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:15.404+0000 ea5d640 -1 osd.3 207 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:50:16.794 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:16.507+0000 ea5d640 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:17.077 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:16.721+0000 ea5d640 -1 osd.3 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4605.0:2 73.1b 73:d96a01e0:::testobj:head [writefull 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66) 2024-05-04T17:50:17.759 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:17.708+0000 ea5d640 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:19.159 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:19.086+0000 ea5d640 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:48:26.630896+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:20.183 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:20.128+0000 ea5d640 -1 osd.3 209 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:50:19.529906+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:21.422 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:21.240+0000 ea5d640 -1 osd.3 210 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:50:19.529906+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:22.060 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-05-04T17:50:22.060 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-05-04T17:50:22.060 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-05-04T17:50:22.060 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-05-04T17:50:22.061 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-05-04T17:50:22.061 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-05-04T17:50:22.061 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-05-04T17:50:22.061 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (26442 ms) 2024-05-04T17:50:22.061 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-05-04T17:50:22.061 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (7779 ms) 2024-05-04T17:50:22.061 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (57815 ms) 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (5405 ms) 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (30622 ms) 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (5766 ms) 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (4721 ms) 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (6276 ms) 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-05-04T17:50:22.062 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (2291 ms) 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (6305 ms) 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (16394 ms) 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (6599 ms) 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (12037 ms) 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (8671 ms) 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (28989 ms) 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (107 ms) 2024-05-04T17:50:22.063 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (226220 ms total) 2024-05-04T17:50:22.064 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: 2024-05-04T17:50:22.064 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-05-04T17:50:22.064 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (240194 ms total) 2024-05-04T17:50:22.064 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-05-04T17:50:23.971 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:23.802+0000 ea5d640 -1 osd.3 211 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:50:21.409094+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:25.112 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:25.059+0000 ea5d640 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:50:21.409094+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:26.374 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:26.190+0000 ea5d640 -1 osd.3 212 heartbeat_check: no reply from 172.21.15.186:6806 osd.6 since back 2024-05-04T17:50:21.409094+0000 front 2024-05-04T17:48:31.014730+0000 (oldest deadline 2024-05-04T17:49:53.090776+0000) 2024-05-04T17:50:26.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-05-04T17:50:26.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (762 ms) 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (343 ms) 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (243699 ms) 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (244804 ms total) 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (244804 ms total) 2024-05-04T17:50:26.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-05-04T17:50:26.752 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:26.477+0000 ea5d640 -1 osd.3 212 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4981.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-05-04T17:50:26.962 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:21.891 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:50:27.012 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:21.940 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:50:27.397 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:22.328 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:50:27.737 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:22.668 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:50:27.752 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:22.683 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:50:27.861 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:22.750 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:50:27.878 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:22.768 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:50:28.134 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:23.064 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:50:28.188 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:23.077 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:50:31.733 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:31.596+0000 ea5d640 -1 osd.3 215 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4981.0:2 99.15 99.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-05-04T17:50:34.090 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T17:50:33.899+0000 ea5d640 -1 osd.3 215 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4981.0:2 99.15 99:ad551702:::test:head [create,setxattr key (5) in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186) 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: Running main() from gmock_main.cc 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [----------] Global test environment set-up. 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1270 ms) 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (294 ms) 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (245886 ms) 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:24.830760+0000 mon.a [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2024-05-04T17:50:40.261 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:24.835790+0000 mon.a [WRN] Health check update: 1 slow ops, oldest one blocked for 165 sec, osd.3 has slow ops (SLOW_OPS) 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:25.475231+0000 mon.a [WRN] Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:26.478598+0000 osd.3 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'Xattrssmithi019-39612-3' : 1 ]) 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:26.500606+0000 mon.b [INF] from='client.? 172.21.15.19:0/2591721922' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi019-39485-1"}]: dispatch 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:29.695304+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi019-39485-1"}]: dispatch 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:29.964953+0000 mon.a [WRN] Health check update: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:29.965423+0000 mon.a [WRN] Health check update: 0 slow ops, oldest one blocked for 170 sec, osd.3 has slow ops (SLOW_OPS) 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:30.089674+0000 mon.a [INF] from='client.? 172.21.15.19:0/2744218090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi019-38394-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:28.814152+0000 mon.c [INF] from='client.? 172.21.15.19:0/3287320058' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi019-38795-7"}]: dispatch 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:31.000507+0000 mon.c [INF] from='client.? 172.21.15.19:0/3082181935' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi019-38691-6"}]: dispatch 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:31.155581+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi019-38795-7"}]: dispatch 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:32.493499+0000 mon.b [INF] from='client.? 172.21.15.19:0/2404596534' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:32.914781+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi019-38691-6"}]: dispatch 2024-05-04T17:50:40.262 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:31.597992+0000 osd.3 [WRN] 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'Xattrssmithi019-39612-3' : 1 ]) 2024-05-04T17:50:40.263 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:32.612311+0000 mon.c [INF] from='client.? 172.21.15.19:0/3287320058' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi019-38795-7"}]: dispatch 2024-05-04T17:50:44.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:34.113484+0000 client.admin [INF api_watch_notify_pp: Running main() from gmock_main.cc 2024-05-04T17:50:44.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-05-04T17:50:44.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: notify 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (22514 ms) 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (68358 ms) 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (90872 ms total) 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: notify 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (9252 ms) 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-05-04T17:50:44.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: notify 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (35535 ms) 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (40453 ms) 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (49457 ms) 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: handle_notify cookie 94211594631072 notify_id 910533066754 notifier_gid 4661 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (1161 ms) 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: handle_notify cookie 94211594631072 notify_id 914828034051 notifier_gid 4661 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (1967 ms) 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: handle_notify cookie 94211594574400 notify_id 927712935938 notifier_gid 4661 2024-05-04T17:50:44.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (12877 ms) 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: handle_notify cookie 94211594574400 notify_id 932007903235 notifier_gid 4661 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (1108 ms) 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: handle_notify cookie 94211594574400 notify_id 932007903236 notifier_gid 4661 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (3416 ms) 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: handle_notify cookie 94211594631072 notify_id 936302870532 notifier_gid 4661 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (408 ms) 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: trying... 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: handle_notify cookie 94211594631072 notify_id 936302870533 notifier_gid 4661 2024-05-04T17:50:44.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: timed out 2024-05-04T17:50:48.519 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_p cmd: Running main() from gmock_main.cc 2024-05-04T17:50:48.519 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [----------] Global test environment set-up. 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (6239 ms) 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (28795 ms) 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (231513 ms) 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [----------] 3 tests from NeoRadosCmd (266547 ms total) 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [----------] Global test environment tear-down 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [==========] 3 tests from 1 test suite ran. (266547 ms total) 2024-05-04T17:50:48.520 INFO:tasks.workunit.client.0.smithi019.stdout: cmd: [ PASSED ] 3 tests. 2024-05-04T17:50:52.156 INFO:tasks.workunit.client.0.smithi019.stdout:] onexx 2024-05-04T17:50:52.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:34.123053+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-05-04T17:50:52.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:34.238804+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi019-39485-1"}]': finished 2024-05-04T17:50:52.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:34.247426+0000 mon.a [INF] from='client.? 172.21.15.19:0/2744218090' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi019-38394-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-05-04T17:50:52.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:34.575929+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi019-38795-7"}]: dispatch 2024-05-04T17:50:52.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:34.988672+0000 mon.a [WRN] Health check update: 0 slow ops, oldest one blocked for 34 sec, osd.3 has slow ops (SLOW_OPS) 2024-05-04T17:50:52.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:35.263245+0000 mon.a [WRN] Health check update: Reduced data availability: 8 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-05-04T17:50:52.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:35.263697+0000 mon.a [WRN] Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-04T17:50:52.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:38.887061+0000 mon.c [INF] from='client.? 172.21.15.19:0/3082181935' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi019-38691-6", "tierpool": "test-rados-api-smithi019-38691-7"}]: dispatch 2024-05-04T17:50:52.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:40.294956+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi019-38691-6", "tierpool": "test-rados-api-smithi019-38691-7"}]: dispatch 2024-05-04T17:50:52.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:40.434298+0000 mon.b [INF] from='client.? 172.21.15.19:0/2404596534' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-05-04T17:50:52.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:40.530435+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi019-39485-1"}]': finished 2024-05-04T17:50:52.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:40.534277+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi019-38795-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-05-04T17:50:52.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:41.545348+0000 mon.a [WRN] Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-04T17:50:52.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:41.553272+0000 mon.a [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 5 pgs peering) 2024-05-04T17:50:52.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:41.554652+0000 mon.a [INF] Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 34 sec, osd.3 has slow ops) 2024-05-04T17:50:52.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:42.054049+0000 client.admin [INF] twoxx 2024-05-04T17:50:52.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:42.063257+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-05-04T17:50:52.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:42.055201+0000 mon.c [INF] from='client.? 172.21.15.19:0/3287320058' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi019-38795-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi019-38795-7"}]: dispatch 2024-05-04T17:50:52.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:43.229917+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi019-38795-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi019-38795-7"}]: dispatch 2024-05-04T17:50:58.720 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:43.347959+0000 mon.b [INF] from='client.? 172.21.15.19:0/1628746822' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi019-38866-7"}]: dispatch 2024-05-04T17:50:58.720 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:43.387479+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi019-38691-6", "tierpool": "test-rados-api-smithi019-38691-7"}]': finished 2024-05-04T17:50:58.720 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:43.450914+0000 mon.b [INF] from='client.? 172.21.15.19:0/445042236' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi019-38841-12"}]: dispatch 2024-05-04T17:50:58.720 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:44.868463+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi019-38866-7"}]: dispatch 2024-05-04T17:50:58.720 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:44.873803+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi019-38841-12"}]: dispatch 2024-05-04T17:50:58.720 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:44.922518+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-05-04T17:50:58.720 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:45.250994+0000 mon.c [INF] from='client.? 172.21.15.19:0/1707688515' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi019-38498-10"}]: dispatch 2024-05-04T17:50:58.720 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:46.107342+0000 mon.b [INF] from='client.? 172.21.15.19:0/138483803' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi019-39485-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-05-04T17:50:58.720 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:46.495481+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi019-38498-10"}]: dispatch 2024-05-04T17:50:58.721 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:47.553450+0000 mon.a [WRN] Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-04T17:50:58.721 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:47.936585+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi019-39485-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-05-04T17:50:58.721 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:47.824224+0000 mon.b [INF] from='client.? 172.21.15.19:0/445042236' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi019-38841-12"}]: dispatch 2024-05-04T17:50:58.721 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:48.634690+0000 mon.b [INF] from='client.? 172.21.15.19:0/2404596534' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-05-04T17:50:58.721 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:52.500552+0000 client.admin [INF] threexx 2024-05-04T17:50:58.721 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:52.518717+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi019-38841-12"}]: dispatch 2024-05-04T17:50:58.721 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:52.530645+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-05-04T17:50:58.721 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:52.577324+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi019-38795-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi019-38795-7"}]': finished 2024-05-04T17:51:02.450 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:52.581104+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi019-38866-7"}]' api_lock_pp: Running main() from gmock_main.cc 2024-05-04T17:51:02.498 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-05-04T17:51:02.498 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-05-04T17:51:02.498 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-05-04T17:51:02.498 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: seed 38498 2024-05-04T17:51:02.498 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-05-04T17:51:02.498 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (15004 ms) 2024-05-04T17:51:02.498 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-05-04T17:51:02.498 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (2534 ms) 2024-05-04T17:51:02.498 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-05-04T17:51:02.499 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (6078 ms) 2024-05-04T17:51:02.499 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-05-04T17:51:02.499 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (65725 ms) 2024-05-04T17:51:02.499 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-05-04T17:51:02.499 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (5253 ms) 2024-05-04T17:51:02.499 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-05-04T17:51:02.499 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (14661 ms) 2024-05-04T17:51:02.499 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-05-04T17:51:02.499 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (19592 ms) 2024-05-04T17:51:02.499 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-05-04T17:51:02.499 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (10873 ms) 2024-05-04T17:51:02.499 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (139720 ms total) 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (58731 ms) 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (29467 ms) 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (5277 ms) 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (3771 ms) 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (7069 ms) 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (494 ms) 2024-05-04T17:51:02.500 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-05-04T17:51:02.501 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (2553 ms) 2024-05-04T17:51:02.501 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-05-04T17:51:02.501 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (166 ms) 2024-05-04T17:51:02.501 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (107528 ms total) 2024-05-04T17:51:02.501 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: 2024-05-04T17:51:02.501 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-05-04T17:51:02.501 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (280871 ms total) 2024-05-04T17:51:02.501 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-05-04T17:51:02.731 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-05-04T17:51:02.731 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-05-04T17:51:02.731 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-05-04T17:51:02.731 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-05-04T17:51:02.731 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: seed 38866 2024-05-04T17:51:02.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-05-04T17:51:02.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (25424 ms) 2024-05-04T17:51:02.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-05-04T17:51:02.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (50014 ms) 2024-05-04T17:51:02.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-05-04T17:51:02.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (25 ms) 2024-05-04T17:51:02.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-05-04T17:51:02.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (1657 ms) 2024-05-04T17:51:02.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-05-04T17:51:02.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (73740 ms) 2024-05-04T17:51:02.732 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (150861 ms total) 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (63102 ms) 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (77 ms) 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (19293 ms) 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (8128 ms) 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (90601 ms total) 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-05-04T17:51:02.733 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (280981 ms total) 2024-05-04T17:51:02.734 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-05-04T17:51:02.796 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:57.726 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:02.816 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:57.746 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:03.085 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:57.974 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:03.115 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:58.004 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:03.232 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:06:58.121 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:03.406 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:06:58.336 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:04.118 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:59.048 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:04.138 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:59.068 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:04.279 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:06:59.210 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:06.959 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-05-04T17:51:06.959 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [----------] Global test environment set-up. 2024-05-04T17:51:06.959 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [----------] 7 tests from LibRadosList 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ OK ] LibRadosList.ListObjects (255983 ms) 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (6669 ms) 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: myset foo1,foo2,foo3 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo1 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo2 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo3 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: myset foo1,foo4,foo5 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo4 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo5 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo1 2024-05-04T17:51:06.960 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: myset foo6,foo7 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo7 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo6 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: ns1:foo4 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: ns1:foo5 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: ns2:foo7 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: ns2:foo6 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: ns1:foo1 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: :foo1 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: :foo2 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: :foo3 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (4157 ms) 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-05-04T17:51:06.961 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 1 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 10 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 13 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 7 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 14 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 0 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 15 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 11 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 5 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 8 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 6 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 3 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 4 0 2024-05-04T17:51:06.962 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 12 0 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 9 0 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 2 0 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (4300 ms) 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: x cursor=MIN 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=1 cursor=5:02547ec2:::1:head 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=10 cursor=5:52ea6a34:::10:head 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=13 cursor=5:566253c9:::13:head 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=7 cursor=5:5c6b0b28:::7:head 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=14 cursor=5:62a1935d:::14:head 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=0 cursor=5:6cac518f:::0:head 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=15 cursor=5:863748b0:::15:head 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=11 cursor=5:89d3ae78:::11:head 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=5 cursor=5:b29083e3:::5:head 2024-05-04T17:51:06.963 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=8 cursor=5:bd63b0f1:::8:head 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=6 cursor=5:c4fdafeb:::6:head 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=3 cursor=5:cfc208b3:::3:head 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=4 cursor=5:d83876eb:::4:head 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=12 cursor=5:de5d7c5f:::12:head 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=9 cursor=5:e960b815:::9:head 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > oid=2 cursor=5:f905c69b:::2:head 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: FIRST> seek to MIN oid=1 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=1 cursor=5:02547ec2:::1:head 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:02547ec2:::1:head 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:02547ec2:::1:head -> 1 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=10 cursor=5:52ea6a34:::10:head 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:52ea6a34:::10:head 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:52ea6a34:::10:head -> 10 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=13 cursor=5:566253c9:::13:head 2024-05-04T17:51:06.964 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:566253c9:::13:head 2024-05-04T17:51:06.965 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:566253c9:::13:head -> 13 2024-05-04T17:51:06.965 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=7 cursor=5:5c6b0b28:::7:head 2024-05-04T17:51:06.965 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:5c6b0b28:::7:head 2024-05-04T17:51:06.965 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:5c6b0b28:::7:head -> 7 2024-05-04T17:51:09.782 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=14 cursor=5:62a1935d:::14: finished 2024-05-04T17:51:09.782 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:52.583774+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi019-38498-10"}]': finished 2024-05-04T17:51:09.782 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: got: 2024-05-04T17:50:53.603911+0000 mon.a [WRN] Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-04T17:51:09.782 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (40513 ms) 2024-05-04T17:51:09.782 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (287964 ms total) 2024-05-04T17:51:09.783 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: 2024-05-04T17:51:09.783 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [----------] Global test environment tear-down 2024-05-04T17:51:09.783 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (287964 ms total) 2024-05-04T17:51:09.783 INFO:tasks.workunit.client.0.smithi019.stdout: api_cmd: [ PASSED ] 4 tests. 2024-05-04T17:51:11.299 INFO:tasks.workunit.client.0.smithi019.stdout::head 2024-05-04T17:51:11.299 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:62a1935d:::14:head 2024-05-04T17:51:11.299 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:62a1935d:::14:head -> 14 2024-05-04T17:51:11.299 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=0 cursor=5:6cac518f:::0:head 2024-05-04T17:51:11.299 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:6cac518f:::0:head 2024-05-04T17:51:11.299 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:6cac518f:::0:head -> 0 2024-05-04T17:51:11.299 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=15 cursor=5:863748b0:::15:head 2024-05-04T17:51:11.299 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:863748b0:::15:head 2024-05-04T17:51:11.299 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:863748b0:::15:head -> 15 2024-05-04T17:51:11.299 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=11 cursor=5:89d3ae78:::11:head 2024-05-04T17:51:11.299 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:89d3ae78:::11:head 2024-05-04T17:51:11.299 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:89d3ae78:::11:head -> 11 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=5 cursor=5:b29083e3:::5:head 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:b29083e3:::5:head 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:b29083e3:::5:head -> 5 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=8 cursor=5:bd63b0f1:::8:head 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:bd63b0f1:::8:head 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:bd63b0f1:::8:head -> 8 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=6 cursor=5:c4fdafeb:::6:head 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:c4fdafeb:::6:head 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:c4fdafeb:::6:head -> 6 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=3 cursor=5:cfc208b3:::3:head 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:cfc208b3:::3:head 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:cfc208b3:::3:head -> 3 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=4 cursor=5:d83876eb:::4:head 2024-05-04T17:51:11.300 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:d83876eb:::4:head 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:d83876eb:::4:head -> 4 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=12 cursor=5:de5d7c5f:::12:head 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:de5d7c5f:::12:head 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:de5d7c5f:::12:head -> 12 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=9 cursor=5:e960b815:::9:head 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:e960b815:::9:head 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:e960b815:::9:head -> 9 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : oid=2 cursor=5:f905c69b:::2:head 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:f905c69b:::2:head 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:f905c69b:::2:head -> 2 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:52ea6a34:::10:head 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:52ea6a34:::10:head expected=5:52ea6a34:::10:head 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:52ea6a34:::10:head -> 10 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=10 expected=10 2024-05-04T17:51:11.301 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:566253c9:::13:head 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:566253c9:::13:head expected=5:566253c9:::13:head 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:566253c9:::13:head -> 13 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=13 expected=13 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:5c6b0b28:::7:head 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:5c6b0b28:::7:head expected=5:5c6b0b28:::7:head 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:5c6b0b28:::7:head -> 7 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=7 expected=7 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:f905c69b:::2:head 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:f905c69b:::2:head expected=5:f905c69b:::2:head 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:f905c69b:::2:head -> 2 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=2 expected=2 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:e960b815:::9:head 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:e960b815:::9:head expected=5:e960b815:::9:head 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:e960b815:::9:head -> 9 2024-05-04T17:51:11.302 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=9 expected=9 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:de5d7c5f:::12:head 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:de5d7c5f:::12:head expected=5:de5d7c5f:::12:head 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:de5d7c5f:::12:head -> 12 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=12 expected=12 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:89d3ae78:::11:head 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:89d3ae78:::11:head expected=5:89d3ae78:::11:head 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:89d3ae78:::11:head -> 11 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=11 expected=11 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:6cac518f:::0:head 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:6cac518f:::0:head expected=5:6cac518f:::0:head 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:6cac518f:::0:head -> 0 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=0 expected=0 2024-05-04T17:51:11.303 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:62a1935d:::14:head 2024-05-04T17:51:22.373 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: Running main() from gmock_main.cc 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [----------] Global test environment set-up. 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (81636 ms) 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLock.LockShared (66800 ms) 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (4513 ms) 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (5642 ms) 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (8514 ms) 2024-05-04T17:51:22.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLock.Unlock (5589 ms) 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (17236 ms) 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (2504 ms) 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [----------] 8 tests from LibRadosLock (192434 ms total) 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (37701 ms) 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (18785 ms) 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (3196 ms) 2024-05-04T17:51:22.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (4898 ms) 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (392 ms) 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (2293 ms) 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (337 ms) 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (190 ms) 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (67792 ms total) 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [----------] Global test environment tear-down 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (300768 ms total) 2024-05-04T17:51:22.376 INFO:tasks.workunit.client.0.smithi019.stdout: api_lock: [ PASSED ] 16 tests. 2024-05-04T17:51:29.220 INFO:tasks.workunit.client.0.smithi019.stdout:p: flushing 2024-05-04T17:51:29.221 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: flushed 2024-05-04T17:51:29.221 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3697 ms) 2024-05-04T17:51:29.221 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-05-04T17:51:29.221 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: trying... 2024-05-04T17:51:29.221 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: handle_notify cookie 94211594574400 notify_id 944892805126 notifier_gid 4661 2024-05-04T17:51:29.221 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: timed out 2024-05-04T17:51:29.221 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: flushing 2024-05-04T17:51:29.221 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: flushed 2024-05-04T17:51:29.221 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (5139 ms) 2024-05-04T17:51:29.221 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-05-04T17:51:29.221 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: List watches 2024-05-04T17:51:29.221 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: notify2 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: handle_notify cookie 94211594574400 notify_id 949187772423 notifier_gid 4661 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: notify2 done 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: watch_check 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: unwatch2 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: flushing 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: done 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3216 ms) 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: List watches 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: notify2 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: handle_notify cookie 94211594574400 notify_id 949187772424 notifier_gid 4661 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: notify2 done 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: watch_check 2024-05-04T17:51:29.222 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: unwatch2 2024-05-04T17:51:29.223 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: flushing 2024-05-04T17:51:29.223 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: done 2024-05-04T17:51:29.223 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3289 ms) 2024-05-04T17:51:29.223 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (170975 ms total) 2024-05-04T17:51:29.223 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: 2024-05-04T17:51:29.223 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-05-04T17:51:29.223 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (307520 ms total) 2024-05-04T17:51:29.223 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-05-04T17:51:30.147 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: Running main() from gmock_main.cc 2024-05-04T17:51:30.147 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-05-04T17:51:30.147 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [----------] Global test environment set-up. 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ OK ] LibRadosStat.Stat (67359 ms) 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (126878 ms) 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ OK ] LibRadosStat.StatNS (21444 ms) 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (192 ms) 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (12697 ms) 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [----------] 5 tests from LibRadosStat (228570 ms total) 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-05-04T17:51:30.148 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (9679 ms) 2024-05-04T17:51:30.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-05-04T17:51:30.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (445 ms) 2024-05-04T17:51:30.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-05-04T17:51:30.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1335 ms) 2024-05-04T17:51:30.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-05-04T17:51:30.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1894 ms) 2024-05-04T17:51:30.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (13353 ms total) 2024-05-04T17:51:30.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: 2024-05-04T17:51:30.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [----------] Global test environment tear-down 2024-05-04T17:51:30.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (308418 ms total) 2024-05-04T17:51:30.149 INFO:tasks.workunit.client.0.smithi019.stdout: api_stat: [ PASSED ] 9 tests. 2024-05-04T17:51:31.123 INFO:tasks.workunit.client.0.smithi019.stdout:0x55a13702a780 2024-05-04T17:51:31.123 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13702b1f0 2024-05-04T17:51:31.123 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137029a10 2024-05-04T17:51:31.123 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13702c0c0 2024-05-04T17:51:31.123 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13702afa0 2024-05-04T17:51:31.123 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13702cfe0 2024-05-04T17:51:31.123 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13702be40 2024-05-04T17:51:31.123 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13702de90 2024-05-04T17:51:31.123 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13702cd10 2024-05-04T17:51:31.123 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13702e9a0 2024-05-04T17:51:31.123 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13702f440 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a13702dc90 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137030330 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137030a00 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137251580 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a1371a40b0 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137051700 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: waiting 0x55a137254ae0 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: done. 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (121267 ms) 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (121267 ms total) 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-05-04T17:51:31.124 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ OK ] LibRadosMisc.Exec (14666 ms) 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (74357 ms) 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (6800 ms) 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5860 ms) 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (80525 ms) 2024-05-04T17:51:31.125 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [----------] 8 tests from LibRadosMisc (182208 ms total) 2024-05-04T17:51:31.126 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: 2024-05-04T17:51:31.126 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [----------] Global test environment tear-down 2024-05-04T17:51:31.126 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (309486 ms total) 2024-05-04T17:51:31.126 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc: [ PASSED ] 12 tests. 2024-05-04T17:51:33.405 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-05-04T17:51:33.405 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-05-04T17:51:33.405 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-05-04T17:51:33.405 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-05-04T17:51:33.405 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-05-04T17:51:33.405 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify_test_cb 2024-05-04T17:51:33.405 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (21840 ms) 2024-05-04T17:51:33.405 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94794371328688 err -107 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (49147 ms) 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94794371330816 err -107 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (24549 ms) 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify2_test_cb from 4531 notify_id 468151435264 cookie 94794371330816 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (36329 ms) 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify2_test_cb from 4531 notify_id 820338753537 cookie 94794371330816 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (98355 ms) 2024-05-04T17:51:33.406 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify2_test_cb from 4531 notify_id 906238099457 cookie 94794371340352 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (2824 ms) 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify2_test_cb from 4531 notify_id 910533066752 cookie 94794371340352 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify2_test_cb from 4531 notify_id 910533066752 cookie 94794372157680 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (354 ms) 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify2_test_cb from 4531 notify_id 910533066753 cookie 94794371340352 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify2_test_cb from 4531 notify_id 914828034050 cookie 94794371340352 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3124 ms) 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94794371340352 err -107 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify2_test_cb from 4531 notify_id 923417968643 cookie 94794371340352 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (6380 ms) 2024-05-04T17:51:33.407 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94794371340352 err -107 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1227 ms) 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (244129 ms total) 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: watch_notify_test_cb 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (13530 ms) 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (13530 ms total) 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-05-04T17:51:33.408 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (311679 ms total) 2024-05-04T17:51:33.409 INFO:tasks.workunit.client.0.smithi019.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-05-04T17:51:36.311 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:07:31.242 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:36.318 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:07:31.249 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:36.380 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:31.269 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:36.382 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:31.272 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:36.396 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:07:31.327 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:36.403 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:07:31.334 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:36.412 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:07:31.343 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:36.478 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:31.367 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:36.514 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:07:31.445 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:56.178 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:07:51.108 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:56.182 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:07:51.113 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:56.263 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:51.152 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:56.266 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:51.155 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:56.274 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:07:51.204 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:56.276 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:07:51.207 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:56.278 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:07:51.209 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:56.356 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:07:51.245 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:56.371 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:07:51.302 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:51:56.592 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: Running main() from gmock_main.cc 2024-05-04T17:51:56.592 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-05-04T17:51:56.592 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [----------] Global test environment set-up. 2024-05-04T17:51:56.592 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-05-04T17:51:56.592 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: seed 38449 2024-05-04T17:51:56.592 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-05-04T17:51:56.592 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (81431 ms) 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (91794 ms) 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (43907 ms) 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (2715 ms) 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (4772 ms) 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (15783 ms) 2024-05-04T17:51:56.593 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (38 ms) 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (5091 ms) 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (82 ms) 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2397 ms) 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (5343 ms) 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (2156 ms) 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-05-04T17:51:56.594 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (275 ms) 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (92 ms) 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (589 ms) 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (250 ms) 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (126 ms) 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (1475 ms) 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (3341 ms) 2024-05-04T17:51:56.595 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (261657 ms total) 2024-05-04T17:51:56.596 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: 2024-05-04T17:51:56.596 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-05-04T17:51:56.596 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-05-04T17:51:56.596 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (6080 ms) 2024-05-04T17:51:56.596 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-05-04T17:51:56.596 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (13747 ms) 2024-05-04T17:51:56.596 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-05-04T17:51:56.596 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (1831 ms) 2024-05-04T17:51:56.596 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-05-04T17:51:56.596 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (1519 ms) 2024-05-04T17:51:56.596 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-05-04T17:52:16.582 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:11.509 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:16.585 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:11.516 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:16.602 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:11.533 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:16.653 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:11.542 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:16.678 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:11.567 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:16.684 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:11.614 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:16.764 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:11.652 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:16.794 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:11.725 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:16.810 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:11.741 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:18.118 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:13.007 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:18.119 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:13.008 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:18.129 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:13.018 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:18.536 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:13.467 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:18.538 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:13.468 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:18.551 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:13.482 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:19.405 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:14.336 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:19.407 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:14.338 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:19.430 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:14.361 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:21.460 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP. list_parallel: process_1_[39202]: starting. 2024-05-04T17:52:21.460 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_1_[39202]: creating pool ceph_test_rados_list_parallel.smithi019-39144 2024-05-04T17:52:21.460 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_1_[39202]: created object 0... 2024-05-04T17:52:21.460 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_1_[39202]: created object 25... 2024-05-04T17:52:21.460 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_1_[39202]: created object 49... 2024-05-04T17:52:21.460 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_1_[39202]: finishing. 2024-05-04T17:52:21.460 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_1_[39202]: shutting down. 2024-05-04T17:52:21.460 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_2_[39203]: starting. 2024-05-04T17:52:21.460 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_2_[39203]: listing objects. 2024-05-04T17:52:21.460 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_2_[39203]: listed object 0... 2024-05-04T17:52:21.460 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_2_[39203]: listed object 25... 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_2_[39203]: saw 50 objects 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_2_[39203]: shutting down. 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_3_[70589]: starting. 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_3_[70589]: creating pool ceph_test_rados_list_parallel.smithi019-39144 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_3_[70589]: created object 0... 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_3_[70589]: created object 25... 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_3_[70589]: created object 49... 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_3_[70589]: finishing. 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_3_[70589]: shutting down. 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_4_[70590]: starting. 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_4_[70590]: listing objects. 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_4_[70590]: listed object 0... 2024-05-04T17:52:21.461 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_4_[70590]: listed object 25... 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_4_[70590]: saw 43 objects 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_4_[70590]: shutting down. 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_5_[70591]: starting. 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_5_[70591]: removed 25 objects... 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_5_[70591]: removed half of the objects 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_5_[70591]: removed 50 objects... 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_5_[70591]: removed 50 objects 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_5_[70591]: shutting down. 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_6_[84592]: starting. 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_6_[84592]: creating pool ceph_test_rados_list_parallel.smithi019-39144 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_6_[84592]: created object 0... 2024-05-04T17:52:21.462 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_6_[84592]: created object 25... 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_6_[84592]: created object 49... 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_6_[84592]: finishing. 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_6_[84592]: shutting down. 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_7_[84593]: starting. 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_7_[84593]: listing objects. 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_7_[84593]: listed object 0... 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_7_[84593]: listed object 25... 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_7_[84593]: listed object 50... 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_7_[84593]: saw 57 objects 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_7_[84593]: shutting down. 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:52:21.463 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:52:21.464 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_8_[84594]: starting. 2024-05-04T17:52:21.464 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_8_[84594]: added 25 objects... 2024-05-04T17:52:21.464 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_8_[84594]: added half of the objects 2024-05-04T17:52:21.464 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_8_[84594]: added 50 objects... 2024-05-04T17:52:21.464 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_8_[84594]: added 50 objects 2024-05-04T17:52:21.464 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_8_[84594]: shutting down. 2024-05-04T17:52:21.464 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:52:21.464 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:52:27.917 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ********************** api_snapshots: Running main() from gmock_main.cc 2024-05-04T17:52:27.917 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-05-04T17:52:27.917 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [----------] Global test environment set-up. 2024-05-04T17:52:27.917 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-05-04T17:52:27.917 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-05-04T17:52:27.917 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (87707 ms) 2024-05-04T17:52:27.917 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (13498 ms) 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (27653 ms) 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (19370 ms) 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (148228 ms total) 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (90304 ms) 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (28107 ms) 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-05-04T17:52:27.918 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (35257 ms) 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (153668 ms total) 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (6566 ms) 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (6461 ms) 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (7462 ms) 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (4606 ms) 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (25095 ms total) 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-05-04T17:52:27.919 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (7143 ms) 2024-05-04T17:52:27.920 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-05-04T17:52:27.920 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (9347 ms) 2024-05-04T17:52:27.920 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (16490 ms total) 2024-05-04T17:52:27.920 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: 2024-05-04T17:52:27.920 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [----------] Global test environment tear-down 2024-05-04T17:52:27.920 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (366221 ms total) 2024-05-04T17:52:27.920 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-05-04T17:52:31.164 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_1_[39209]: starting. 2024-05-04T17:52:31.164 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_1_[39209]: creating pool ceph_test_rados_open_pools_parallel.smithi019-39154 2024-05-04T17:52:31.164 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_1_[39209]: created object 0... 2024-05-04T17:52:31.164 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_1_[39209]: created object 25... 2024-05-04T17:52:31.164 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_1_[39209]: created object 49... 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_1_[39209]: finishing. 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_1_[39209]: shutting down. 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_2_[39210]: starting. 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_2_[39210]: rados_pool_create. 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_2_[39210]: rados_ioctx_create. 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_2_[39210]: shutting down. 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: ******************************* 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_3_[76425]: starting. 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_3_[76425]: creating pool ceph_test_rados_open_pools_parallel.smithi019-39154 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_3_[76425]: created object 0... 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_3_[76425]: created object 25... 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_3_[76425]: created object 49... 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_3_[76425]: finishing. 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_3_[76425]: shutting down. 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: ******************************* 2024-05-04T17:52:31.165 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_4_[76427]: starting. 2024-05-04T17:52:31.166 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_4_[76427]: rados_pool_create. 2024-05-04T17:52:31.166 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_4_[76427]: rados_ioctx_create. 2024-05-04T17:52:31.166 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: process_4_[76427]: shutting down. 2024-05-04T17:52:31.166 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: ******************************* 2024-05-04T17:52:31.166 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: ******************************* 2024-05-04T17:52:31.166 INFO:tasks.workunit.client.0.smithi019.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-05-04T17:52:36.780 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:31.710 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:36.804 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:31.693 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:36.805 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:31.735 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:36.813 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:31.744 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:36.867 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:31.796 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:36.927 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:31.816 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:37.022 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:31.952 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:37.037 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:31.926 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:37.056 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:31.986 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:46.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-05-04T17:52:46.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-05-04T17:52:46.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-05-04T17:52:46.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-05-04T17:52:46.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: seed 38772 2024-05-04T17:52:46.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-05-04T17:52:46.482 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (83384 ms) 2024-05-04T17:52:46.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-05-04T17:52:46.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (17837 ms) 2024-05-04T17:52:46.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-05-04T17:52:46.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (66064 ms) 2024-05-04T17:52:46.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-05-04T17:52:46.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (16133 ms) 2024-05-04T17:52:46.483 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (8964 ms) 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (192382 ms total) 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (12903 ms) 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (47329 ms) 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (44563 ms) 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (12564 ms) 2024-05-04T17:52:46.486 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (5724 ms) 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-2497-gf273489c/rpm/el9/BUILD/ceph-19.0.0-2497-gf273489c/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi019-38772-7 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: waiting for snaps to purge 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (19016 ms) 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (142099 ms total) 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-05-04T17:52:46.487 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (8 ms) 2024-05-04T17:52:46.488 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-05-04T17:52:46.488 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5735 ms) 2024-05-04T17:52:46.488 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5743 ms total) 2024-05-04T17:52:46.488 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: 2024-05-04T17:52:46.488 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-05-04T17:52:46.488 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-05-04T17:52:46.488 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (10333 ms) 2024-05-04T17:52:46.488 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-05-04T17:52:46.489 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK api_io: Running main() from gmock_main.cc 2024-05-04T17:52:46.489 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-05-04T17:52:46.489 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [----------] Global test environment set-up. 2024-05-04T17:52:46.489 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-05-04T17:52:46.489 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-05-04T17:52:46.489 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (87563 ms) 2024-05-04T17:52:46.489 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-05-04T17:52:46.489 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-05-04T17:52:46.489 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-05-04T17:52:46.489 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (13682 ms) 2024-05-04T17:52:46.489 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-05-04T17:52:46.489 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (8680 ms) 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.Checksum (99550 ms) 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (42561 ms) 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (3568 ms) 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (105 ms) 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (34 ms) 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.TruncTest (160 ms) 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (217 ms) 2024-05-04T17:52:46.490 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (1869 ms) 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.RmXattr (3591 ms) 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIo.XattrIter (2050 ms) 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [----------] 14 tests from LibRadosIo (263630 ms total) 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (8951 ms) 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (4214 ms) 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (3045 ms) 2024-05-04T17:52:46.491 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-05-04T17:52:46.492 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (3927 ms) 2024-05-04T17:52:46.492 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-05-04T17:52:46.492 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (3519 ms) 2024-05-04T17:52:46.492 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-05-04T17:52:46.492 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (1455 ms) 2024-05-04T17:52:46.492 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-05-04T17:52:46.492 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (1931 ms) 2024-05-04T17:52:46.493 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-05-04T17:52:46.493 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (2564 ms) 2024-05-04T17:52:46.493 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-05-04T17:52:46.493 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (40443 ms) 2024-05-04T17:52:46.493 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-05-04T17:52:46.493 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (3645 ms) 2024-05-04T17:52:46.493 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [----------] 10 tests from LibRadosIoEC (73694 ms total) 2024-05-04T17:52:46.493 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: 2024-05-04T17:52:46.493 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [----------] Global test environment tear-down 2024-05-04T17:52:46.493 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [==========] 24 tests from 2 test suites ran. (384944 ms total) 2024-05-04T17:52:46.493 INFO:tasks.workunit.client.0.smithi019.stdout: api_io: [ PASSED ] 24 tests. 2024-05-04T17:52:56.968 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:51.899 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:57.036 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:51.925 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:57.039 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:51.969 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:57.041 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:51.930 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:57.106 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:52.037 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:57.116 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:52.047 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:57.134 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:08:52.023 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:57.204 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:08:52.135 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:52:57.277 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:08:52.208 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:17.263 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:12.193 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:17.268 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:12.198 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:17.334 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:12.223 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:17.336 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:12.225 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:17.350 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:12.280 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:17.357 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:12.288 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:17.365 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:12.296 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:17.430 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:12.319 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:17.441 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:12.372 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:18.607 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: Running main() from gmock_main.cc 2024-05-04T17:53:18.607 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [----------] Global test environment set-up. 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (45972 ms) 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (18402 ms) 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (78398 ms) 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (15111 ms) 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (30682 ms) 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (188565 ms total) 2024-05-04T17:53:18.608 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (72345 ms) 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (48112 ms) 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (28898 ms) 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (13613 ms) 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (42143 ms) 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi019-39593-11. 2024-05-04T17:53:18.609 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: Waiting for snaps to purge. 2024-05-04T17:53:18.610 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (22859 ms) 2024-05-04T17:53:18.610 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (227970 ms total) 2024-05-04T17:53:18.610 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: 2024-05-04T17:53:18.610 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [----------] Global test environment tear-down 2024-05-04T17:53:18.610 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (416535 ms total) 2024-05-04T17:53:18.610 INFO:tasks.workunit.client.0.smithi019.stdout: snapshots: [ PASSED ] 11 tests. 2024-05-04T17:53:30.052 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: Running main() from gmock_main.cc 2024-05-04T17:53:30.052 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-05-04T17:53:30.052 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [----------] Global test environment set-up. 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (57396 ms) 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (202926 ms) 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (30986 ms) 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (29493 ms) 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (17096 ms) 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (18407 ms) 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-05-04T17:53:30.053 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (5530 ms) 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (6105 ms) 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (6466 ms) 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (7667 ms) 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (22008 ms) 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (8169 ms) 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (10222 ms) 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (5515 ms) 2024-05-04T17:53:30.054 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (427986 ms total) 2024-05-04T17:53:30.055 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: 2024-05-04T17:53:30.055 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [----------] Global test environment tear-down 2024-05-04T17:53:30.055 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (427986 ms total) 2024-05-04T17:53:30.055 INFO:tasks.workunit.client.0.smithi019.stdout: read_operations: [ PASSED ] 14 tests. 2024-05-04T17:53:34.604 INFO:tasks.workunit.client.0.smithi019.stdout: io: Running main() from gmock_main.cc 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [----------] Global test environment set-up. 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [----------] 14 tests from NeoRadosIo 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.Limits (77377 ms) 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (139009 ms) 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.ReadOp (31576 ms) 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.SparseRead (28087 ms) 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.RoundTrip (33064 ms) 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-05-04T17:53:34.605 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (13058 ms) 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (10336 ms) 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (30627 ms) 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (9188 ms) 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.Trunc (9821 ms) 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.Remove (7242 ms) 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (4367 ms) 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-05-04T17:53:34.606 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.RmXattr (20758 ms) 2024-05-04T17:53:34.607 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-05-04T17:53:34.607 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ OK ] NeoRadosIo.GetXattrs (18103 ms) 2024-05-04T17:53:34.607 INFO:tasks.workunit.client.0.smithi019.stdout: io: [----------] 14 tests from NeoRadosIo (432613 ms total) 2024-05-04T17:53:34.607 INFO:tasks.workunit.client.0.smithi019.stdout: io: 2024-05-04T17:53:34.607 INFO:tasks.workunit.client.0.smithi019.stdout: io: [----------] Global test environment tear-down 2024-05-04T17:53:34.607 INFO:tasks.workunit.client.0.smithi019.stdout: io: [==========] 14 tests from 1 test suite ran. (432613 ms total) 2024-05-04T17:53:34.607 INFO:tasks.workunit.client.0.smithi019.stdout: io: [ PASSED ] 14 tests. 2024-05-04T17:53:37.453 INFO:tasks.workunit.client.0.smithi019.stdout:RoundTripPP2 (3583 ms) 2024-05-04T17:53:37.453 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (3748 ms) 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (6209 ms) 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (4258 ms) 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (334 ms) 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (1384 ms) 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (1783 ms) 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (31212 ms) 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-05-04T17:53:37.454 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (5621 ms) 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (17382 ms) 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (9064 ms) 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (1400 ms) 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (15649 ms) 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (124804 ms total) 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [----------] Global test environment tear-down 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (435909 ms total) 2024-05-04T17:53:37.455 INFO:tasks.workunit.client.0.smithi019.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-05-04T17:53:37.494 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:32.423 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:37.552 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:32.441 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:37.556 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:32.445 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:37.568 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:32.499 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:37.620 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:32.509 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:37.662 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:32.592 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:37.676 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:32.606 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:37.678 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:32.609 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:37.773 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:32.704 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:53:42.826 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_1_[39350]: starting. 2024-05-04T17:53:42.826 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_1_[39350]: creating pool ceph_test_rados_delete_pools_parallel.smithi019-39278 2024-05-04T17:53:42.826 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_1_[39350]: created object 0... 2024-05-04T17:53:42.826 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_1_[39350]: created object 25... 2024-05-04T17:53:42.826 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_1_[39350]: created object 49... 2024-05-04T17:53:42.826 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_1_[39350]: finishing. 2024-05-04T17:53:42.826 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_1_[39350]: shutting down. 2024-05-04T17:53:42.826 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_2_[39351]: starting. 2024-05-04T17:53:42.826 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_2_[39351]: deleting pool ceph_test_rados_delete_pools_parallel.smithi019-39278 2024-05-04T17:53:42.826 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_2_[39351]: shutting down. 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: ******************************* 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_3_[79582]: starting. 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_3_[79582]: creating pool ceph_test_rados_delete_pools_parallel.smithi019-39278 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_3_[79582]: created object 0... 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_3_[79582]: created object 25... 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_3_[79582]: created object 49... 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_3_[79582]: finishing. 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_3_[79582]: shutting down. 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: ******************************* 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_5_[79584]: starting. 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_5_[79584]: listing objects. 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_5_[79584]: listed object 0... 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_5_[79584]: listed object 25... 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_5_[79584]: saw 50 objects 2024-05-04T17:53:42.827 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_5_[79584]: shutting down. 2024-05-04T17:53:42.828 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: ******************************* 2024-05-04T17:53:42.828 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_4_[79583]: starting. 2024-05-04T17:53:42.828 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_4_[79583]: deleting pool ceph_test_rados_delete_pools_parallel.smithi019-39278 2024-05-04T17:53:42.828 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: process_4_[79583]: shutting down. 2024-05-04T17:53:42.828 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: ******************************* 2024-05-04T17:53:42.828 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: ******************************* 2024-05-04T17:53:42.828 INFO:tasks.workunit.client.0.smithi019.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-05-04T17:53:43.976 INFO:tasks.workunit.client.0.smithi019.stdout: cls: Running main() from gmock_main.cc 2024-05-04T17:53:43.976 INFO:tasks.workunit.client.0.smithi019.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-05-04T17:53:43.977 INFO:tasks.workunit.client.0.smithi019.stdout: cls: [----------] Global test environment set-up. 2024-05-04T17:53:43.977 INFO:tasks.workunit.client.0.smithi019.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-05-04T17:53:43.977 INFO:tasks.workunit.client.0.smithi019.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-05-04T17:53:43.977 INFO:tasks.workunit.client.0.smithi019.stdout: cls: [ OK ] NeoRadosCls.DNE (430227 ms) 2024-05-04T17:53:43.977 INFO:tasks.workunit.client.0.smithi019.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-05-04T17:53:43.977 INFO:tasks.workunit.client.0.smithi019.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (11801 ms) 2024-05-04T17:53:43.977 INFO:tasks.workunit.client.0.smithi019.stdout: cls: [----------] 2 tests from NeoRadosCls (442028 ms total) 2024-05-04T17:53:43.977 INFO:tasks.workunit.client.0.smithi019.stdout: cls: 2024-05-04T17:53:43.977 INFO:tasks.workunit.client.0.smithi019.stdout: cls: [----------] Global test environment tear-down 2024-05-04T17:53:43.977 INFO:tasks.workunit.client.0.smithi019.stdout: cls: [==========] 2 tests from 1 test suite ran. (442028 ms total) 2024-05-04T17:53:43.977 INFO:tasks.workunit.client.0.smithi019.stdout: cls: [ PASSED ] 2 tests. 2024-05-04T17:54:02.409 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:57.339 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:02.494 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:57.425 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:02.546 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:57.435 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:02.549 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:57.438 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:02.579 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:57.510 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:02.581 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:57.512 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:02.613 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:09:57.502 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:02.640 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:09:57.570 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:02.702 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:09:57.632 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:03.373 INFO:tasks.workunit.client.0.smithi019.stdout:] LibRadosSnapshotsECPP.SnapRemovePP (20292 ms) 2024-05-04T17:54:03.373 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (27838 ms) 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (11524 ms) 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (69987 ms total) 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (14478 ms) 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (6996 ms) 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (5962 ms) 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (27436 ms total) 2024-05-04T17:54:03.374 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: 2024-05-04T17:54:03.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-05-04T17:54:03.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (461683 ms total) 2024-05-04T17:54:03.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-05-04T17:54:03.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-05-04T17:54:03.375 INFO:tasks.workunit.client.0.smithi019.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-05-04T17:54:05.760 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: Running main() from gmock_main.cc 2024-05-04T17:54:05.760 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-05-04T17:54:05.760 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [----------] Global test environment set-up. 2024-05-04T17:54:05.760 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-05-04T17:54:05.760 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-05-04T17:54:05.760 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.TooBig (5094 ms) 2024-05-04T17:54:05.760 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-05-04T17:54:05.760 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (65383 ms) 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (50823 ms) 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (27121 ms) 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (7711 ms) 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (13463 ms) 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (14024 ms) 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (54503 ms) 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (22517 ms) 2024-05-04T17:54:05.761 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (30953 ms) 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (30952 ms) 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (13873 ms) 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (6672 ms) 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (24433 ms) 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.Flush (4221 ms) 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (4986 ms) 2024-05-04T17:54:05.762 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (23160 ms) 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (5737 ms) 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (4316 ms) 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (5634 ms) 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (19578 ms) 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (6137 ms) 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (6769 ms) 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-05-04T17:54:05.763 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (6192 ms) 2024-05-04T17:54:05.764 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-05-04T17:54:05.764 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (4996 ms) 2024-05-04T17:54:05.764 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-05-04T17:54:05.764 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (4982 ms) 2024-05-04T17:54:05.764 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [----------] 26 tests from LibRadosAio (464233 ms total) 2024-05-04T17:54:05.764 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: 2024-05-04T17:54:05.764 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-05-04T17:54:07.909 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC********* 2024-05-04T17:54:07.909 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_9_[97346]: starting. 2024-05-04T17:54:07.909 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_9_[97346]: creating pool ceph_test_rados_list_parallel.smithi019-39144 2024-05-04T17:54:07.909 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_9_[97346]: created object 0... 2024-05-04T17:54:07.909 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_9_[97346]: created object 25... 2024-05-04T17:54:07.909 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_9_[97346]: created object 49... 2024-05-04T17:54:07.909 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_9_[97346]: finishing. 2024-05-04T17:54:07.909 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_9_[97346]: shutting down. 2024-05-04T17:54:07.909 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.909 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.909 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_10_[97347]: starting. 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_10_[97347]: listing objects. 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_10_[97347]: listed object 0... 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_10_[97347]: listed object 25... 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_10_[97347]: listed object 50... 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_10_[97347]: listed object 75... 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_10_[97347]: listed object 100... 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_10_[97347]: saw 105 objects 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_10_[97347]: shutting down. 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_13_[97350]: starting. 2024-05-04T17:54:07.910 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_13_[97350]: removed 25 objects... 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_13_[97350]: removed half of the objects 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_13_[97350]: removed 50 objects... 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_13_[97350]: removed 50 objects 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_13_[97350]: shutting down. 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_11_[97348]: starting. 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_11_[97348]: added 25 objects... 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_11_[97348]: added half of the objects 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_11_[97348]: added 50 objects... 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_11_[97348]: added 50 objects 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_11_[97348]: shutting down. 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.911 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_12_[97349]: starting. 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_12_[97349]: added 25 objects... 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_12_[97349]: added half of the objects 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_12_[97349]: added 50 objects... 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_12_[97349]: added 50 objects 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_12_[97349]: shutting down. 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_14_[104055]: starting. 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_14_[104055]: creating pool ceph_test_rados_list_parallel.smithi019-39144 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_14_[104055]: created object 0... 2024-05-04T17:54:07.912 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_14_[104055]: created object 25... 2024-05-04T17:54:07.913 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_14_[104055]: created object 49... 2024-05-04T17:54:07.913 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_14_[104055]: finishing. 2024-05-04T17:54:07.913 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_14_[104055]: shutting down. 2024-05-04T17:54:07.913 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.913 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.913 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.913 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:07.913 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_15_[104056]: starting. 2024-05-04T17:54:07.913 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_15_[104056]: listing objects. 2024-05-04T17:54:10.137 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:05.067 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:10.218 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:05.149 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:10.227 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:05.116 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:10.231 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:05.120 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:10.254 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:05.184 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:10.262 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:05.193 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:10.286 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:05.175 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:10.302 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:05.233 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:10.347 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:05.278 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:14.839 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_15_[104056]: listed object 0... 2024-05-04T17:54:14.839 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_15_[104056]: listed object 25... 2024-05-04T17:54:14.839 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_15_[104056]: listed object 50... 2024-05-04T17:54:14.839 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_15_[104056]: listed object 75... 2024-05-04T17:54:14.839 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_15_[104056]: listed object 100... 2024-05-04T17:54:14.839 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_15_[104056]: listed object 125... 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_15_[104056]: saw 150 objects 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_15_[104056]: shutting down. 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_16_[104057]: starting. 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_16_[104057]: added 25 objects... 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_16_[104057]: added half of the objects 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_16_[104057]: added 50 objects... 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_16_[104057]: added 50 objects 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: process_16_[104057]: shutting down. 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:14.840 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:14.841 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:14.841 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:14.841 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******************************* 2024-05-04T17:54:14.841 INFO:tasks.workunit.client.0.smithi019.stdout: list_parallel: ******* SUCCESS ********** 2024-05-04T17:54:22.674 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:17.605 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:22.694 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:17.625 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:22.743 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:17.632 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:22.757 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:17.646 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:22.790 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:17.721 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:22.805 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:17.736 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:22.809 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:17.739 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:22.820 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:17.709 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:22.892 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:17.822 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:42.910 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:37.799 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:42.963 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:37.852 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:42.968 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:37.899 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:42.978 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:37.909 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:43.054 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:37.985 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:43.060 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:37.950 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:43.061 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:37.992 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:43.101 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:38.032 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:43.133 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:38.064 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:54:52.576 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-05-04T17:54:52.576 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-05-04T17:54:52.576 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-05-04T17:54:52.576 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (6678 ms) 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (368754 ms) 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (16679 ms) 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (7769 ms) 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (5713 ms) 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (4287 ms) 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (21958 ms) 2024-05-04T17:54:52.577 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (4118 ms) 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (5417 ms) 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (5639 ms) 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3532 ms) 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2555 ms) 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3951 ms) 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (4719 ms) 2024-05-04T17:54:52.578 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (4683 ms) 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3760 ms) 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (4347 ms) 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3564 ms) 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (4083 ms) 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3676 ms) 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3412 ms) 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-05-04T17:54:52.579 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3365 ms) 2024-05-04T17:54:52.580 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-05-04T17:54:52.580 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3559 ms) 2024-05-04T17:54:52.580 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-05-04T17:54:52.580 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (4529 ms) 2024-05-04T17:54:52.580 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-05-04T17:54:52.580 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3363 ms) 2024-05-04T17:54:52.580 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-05-04T17:54:52.580 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3475 ms) 2024-05-04T17:54:52.580 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-05-04T17:55:02.937 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:57.867 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:02.940 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:57.871 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:03.008 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:57.938 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:03.008 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:10:57.939 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:03.011 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:57.900 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:03.019 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:57.950 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:03.023 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:57.913 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:03.084 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:10:57.973 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:03.129 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:10:58.038 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:14.246 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRados list: Running main() from gmock_main.cc 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [----------] Global test environment set-up. 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [----------] 3 tests from NeoradosList 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [ OK ] NeoradosList.ListObjects (217988 ms) 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [ OK ] NeoradosList.ListObjectsNS (57227 ms) 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [ OK ] NeoradosList.ListObjectsMany (257007 ms) 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [----------] 3 tests from NeoradosList (532222 ms total) 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [----------] Global test environment tear-down 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [==========] 3 tests from 1 test suite ran. (532222 ms total) 2024-05-04T17:55:14.247 INFO:tasks.workunit.client.0.smithi019.stdout: list: [ PASSED ] 3 tests. 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (81683 ms) 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (34548 ms) 2024-05-04T17:55:15.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (17939 ms) 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (13974 ms) 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (12271 ms) 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (59888 ms) 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (313246 ms) 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (533549 ms total) 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (533550 ms total) 2024-05-04T17:55:15.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-05-04T17:55:19.466 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:62a1935d:::14:head expected=5:62a1935d:::14:head 2024-05-04T17:55:19.467 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:62a1935d:::14:head -> 14 2024-05-04T17:55:19.467 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=14 expected=14 2024-05-04T17:55:19.467 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:02547ec2:::1:head 2024-05-04T17:55:19.467 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:02547ec2:::1:head expected=5:02547ec2:::1:head 2024-05-04T17:55:19.467 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:02547ec2:::1:head -> 1 2024-05-04T17:55:19.467 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=1 expected=1 2024-05-04T17:55:19.467 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:d83876eb:::4:head 2024-05-04T17:55:19.467 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:d83876eb:::4:head expected=5:d83876eb:::4:head 2024-05-04T17:55:19.467 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:d83876eb:::4:head -> 4 2024-05-04T17:55:19.467 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=4 expected=4 2024-05-04T17:55:19.467 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:cfc208b3:::3:head 2024-05-04T17:55:19.468 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:cfc208b3:::3:head expected=5:cfc208b3:::3:head 2024-05-04T17:55:19.468 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:cfc208b3:::3:head -> 3 2024-05-04T17:55:19.468 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=3 expected=3 2024-05-04T17:55:19.468 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:c4fdafeb:::6:head 2024-05-04T17:55:19.468 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:c4fdafeb:::6:head expected=5:c4fdafeb:::6:head 2024-05-04T17:55:19.468 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:c4fdafeb:::6:head -> 6 2024-05-04T17:55:19.468 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=6 expected=6 2024-05-04T17:55:19.468 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:bd63b0f1:::8:head 2024-05-04T17:55:19.468 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:bd63b0f1:::8:head expected=5:bd63b0f1:::8:head 2024-05-04T17:55:19.468 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:bd63b0f1:::8:head -> 8 2024-05-04T17:55:19.468 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=8 expected=8 2024-05-04T17:55:19.468 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:b29083e3:::5:head 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:b29083e3:::5:head expected=5:b29083e3:::5:head 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:b29083e3:::5:head -> 5 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=5 expected=5 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : seek to 5:863748b0:::15:head 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : cursor()=5:863748b0:::15:head expected=5:863748b0:::15:head 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: > 5:863748b0:::15:head -> 15 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: : entry=15 expected=15 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (19404 ms) 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (165267 ms) 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: split 0/5 -> MIN 5:33333333::::head 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: split 1/5 -> 5:33333333::::head 5:66666666::::head 2024-05-04T17:55:19.469 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: split 2/5 -> 5:66666666::::head 5:99999999::::head 2024-05-04T17:55:19.470 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: split 3/5 -> 5:99999999::::head 5:cccccccc::::head 2024-05-04T17:55:19.470 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: split 4/5 -> 5:cccccccc::::head MAX 2024-05-04T17:55:19.470 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (55945 ms) 2024-05-04T17:55:19.470 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [----------] 7 tests from LibRadosList (511725 ms total) 2024-05-04T17:55:19.470 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 2024-05-04T17:55:19.470 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-05-04T17:55:19.470 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-05-04T17:55:19.470 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (3020 ms) 2024-05-04T17:55:19.470 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-05-04T17:55:19.470 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: myset foo1,foo2,foo3 2024-05-04T17:55:19.470 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo1 2024-05-04T17:55:19.470 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo2 2024-05-04T17:55:19.471 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo3 2024-05-04T17:55:19.471 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: myset foo1,foo4,foo5 2024-05-04T17:55:19.471 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo4 2024-05-04T17:55:19.471 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo5 2024-05-04T17:55:19.471 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo1 2024-05-04T17:55:19.471 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: myset foo6,foo7 2024-05-04T17:55:19.471 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo7 2024-05-04T17:55:19.471 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: foo6 2024-05-04T17:55:19.471 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-05-04T17:55:19.471 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: ns1:foo4 2024-05-04T17:55:19.471 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: ns1:foo5 2024-05-04T17:55:19.471 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: ns2:foo7 2024-05-04T17:55:19.472 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: ns2:foo6 2024-05-04T17:55:19.472 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: ns1:foo1 2024-05-04T17:55:19.472 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: :foo1 2024-05-04T17:55:19.472 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: :foo2 2024-05-04T17:55:19.472 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: :foo3 2024-05-04T17:55:19.472 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (9893 ms) 2024-05-04T17:55:19.472 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-05-04T17:55:19.472 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 1 0 2024-05-04T17:55:19.472 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 10 0 2024-05-04T17:55:19.472 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 13 0 2024-05-04T17:55:23.027 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:17.957 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:23.037 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:17.968 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:23.089 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:17.978 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:23.091 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:17.980 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:23.109 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:18.040 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:23.112 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:18.043 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:23.141 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:18.071 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:23.145 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:18.034 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:23.191 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:18.122 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:26.564 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: Running main() from gmock_main.cc 2024-05-04T17:55:26.564 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-05-04T17:55:26.564 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [----------] Global test environment set-up. 2024-05-04T17:55:26.565 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-05-04T17:55:26.565 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-05-04T17:55:26.565 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (259771 ms) 2024-05-04T17:55:26.565 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-05-04T17:55:26.565 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (61046 ms) 2024-05-04T17:55:26.565 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-05-04T17:55:26.565 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (223728 ms) 2024-05-04T17:55:26.565 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [----------] 3 tests from NeoradosECList (544545 ms total) 2024-05-04T17:55:26.565 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: 2024-05-04T17:55:26.565 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [----------] Global test environment tear-down 2024-05-04T17:55:26.565 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (544545 ms total) 2024-05-04T17:55:26.565 INFO:tasks.workunit.client.0.smithi019.stdout: ec_list: [ PASSED ] 3 tests. 2024-05-04T17:55:28.174 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 7 0 2024-05-04T17:55:28.174 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 14 0 2024-05-04T17:55:28.174 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 0 0 2024-05-04T17:55:28.174 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 15 0 2024-05-04T17:55:28.174 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 11 0 2024-05-04T17:55:28.174 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 5 0 2024-05-04T17:55:28.174 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 8 0 2024-05-04T17:55:28.174 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 6 0 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 3 0 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 4 0 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 12 0 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 9 0 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 2 0 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (7713 ms) 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [----------] 3 tests from LibRadosListEC (20626 ms total) 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3705 ms) 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [----------] 1 test from LibRadosListNP (3705 ms total) 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: 2024-05-04T17:55:28.175 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [----------] Global test environment tear-down 2024-05-04T17:55:28.176 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [==========] 11 tests from 3 test suites ran. (546596 ms total) 2024-05-04T17:55:28.176 INFO:tasks.workunit.client.0.smithi019.stdout: api_list: [ PASSED ] 11 tests. 2024-05-04T17:55:43.160 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:38.090 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:43.223 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:38.154 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:43.224 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:38.112 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:43.227 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:38.116 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:43.281 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:38.170 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:43.351 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:38.282 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:43.354 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:38.285 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:43.357 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:38.288 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:55:43.428 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:38.358 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:03.309 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:58.239 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:03.313 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:58.244 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:03.365 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:11:58.295 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:03.375 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:58.264 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:03.376 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:58.265 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:03.384 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:58.315 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:03.387 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:58.318 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:03.426 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:11:58.315 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:03.440 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:11:58.371 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:06.232 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:01.162 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:06.244 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:01.175 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:06.268 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:01.198 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:06.301 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:01.190 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:06.302 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:01.191 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:06.321 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:01.252 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:06.323 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:01.254 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:06.327 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:01.216 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:06.357 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:01.287 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:14.043 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: Running main() from gmock_main.cc 2024-05-04T17:56:14.043 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-05-04T17:56:14.043 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [----------] Global test environment set-up. 2024-05-04T17:56:14.043 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-05-04T17:56:14.043 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-05-04T17:56:14.043 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (83722 ms) 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (113899 ms) 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (64930 ms) 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (39537 ms) 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (14305 ms) 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (7750 ms) 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (267822 ms) 2024-05-04T17:56:14.044 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (591965 ms total) 2024-05-04T17:56:14.045 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: 2024-05-04T17:56:14.045 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [----------] Global test environment tear-down 2024-05-04T17:56:14.045 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (591965 ms total) 2024-05-04T17:56:14.045 INFO:tasks.workunit.client.0.smithi019.stdout: write_operations: [ PASSED ] 7 tests. 2024-05-04T17:56:15.137 INFO:tasks.workunit.client.0.smithi019.stdout:.SimpleWrite 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (8268 ms) 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (8016 ms) 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (9952 ms) 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (8039 ms) 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7848 ms) 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (8958 ms) 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7996 ms) 2024-05-04T17:56:15.138 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (8234 ms) 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (8041 ms) 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (9231 ms) 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7821 ms) 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7480 ms) 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6754 ms) 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (8021 ms) 2024-05-04T17:56:15.139 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-05-04T17:56:15.140 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7365 ms) 2024-05-04T17:56:15.140 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-05-04T17:56:15.140 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7350 ms) 2024-05-04T17:56:15.140 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (129374 ms total) 2024-05-04T17:56:15.140 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: 2024-05-04T17:56:15.140 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [----------] Global test environment tear-down 2024-05-04T17:56:15.140 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (593607 ms total) 2024-05-04T17:56:15.140 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio: [ PASSED ] 42 tests. 2024-05-04T17:56:23.559 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:18.489 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:23.562 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:18.492 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:23.620 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:18.551 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:23.622 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:18.511 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:23.627 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:18.516 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:23.630 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:18.561 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:23.631 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:18.562 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:23.674 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:18.562 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:23.682 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:18.613 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:38.648 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:33.578 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:38.674 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:33.605 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:38.702 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:33.591 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:38.703 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:33.593 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:38.710 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:33.641 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:38.712 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:33.643 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:38.730 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:33.661 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:38.754 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:33.643 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:38.755 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:33.685 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:43.524 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:38.454 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:43.526 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:38.457 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:43.553 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:38.483 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:43.577 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:38.466 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:43.579 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:38.468 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:43.580 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:38.510 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:43.584 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:38.515 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:43.596 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:38.485 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:43.601 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:38.532 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:53.683 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:48.613 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:53.688 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:48.619 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:53.689 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:48.620 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:53.693 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:48.582 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:53.693 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:48.624 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:53.697 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:48.586 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:53.724 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:48.654 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:53.724 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:48.655 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:56:53.725 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:48.614 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:03.904 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:58.834 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:03.913 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:58.844 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:03.965 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:12:58.896 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:03.976 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:58.907 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:03.987 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:58.876 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:03.997 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:58.886 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:03.999 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:58.930 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:04.040 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:12:58.929 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:04.050 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:12:58.981 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:08.761 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:03.691 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:08.763 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:03.693 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:08.778 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:03.709 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:08.814 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:03.744 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:08.815 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:03.746 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:08.816 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:03.705 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:08.817 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:03.706 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:08.831 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:03.761 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:08.832 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:03.721 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:16.324 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-05-04T17:57:16.324 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-05-04T17:57:16.324 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-05-04T17:57:16.324 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: seed 38686 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (201 ms) 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (63789 ms) 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (152547 ms) 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (29907 ms) 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (1199 ms) 2024-05-04T17:57:16.325 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (306 ms) 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (694 ms) 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (365 ms) 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2616 ms) 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (550 ms) 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (45 ms) 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (155 ms) 2024-05-04T17:57:16.326 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (476 ms) 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: osd_max_attr_size = 0 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (305184 ms) 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (9494 ms) 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: done waiting, doing copies 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: waiting for final deep scrubs... 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: done waiting 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (73401 ms) 2024-05-04T17:57:16.327 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-05-04T17:57:16.328 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (96 ms) 2024-05-04T17:57:16.328 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-05-04T17:57:16.328 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (50 ms) 2024-05-04T17:57:16.328 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-05-04T17:57:16.328 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5807 ms) 2024-05-04T17:57:16.328 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-05-04T17:57:16.328 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-05-04T17:57:16.328 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-05-04T17:57:16.328 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-05-04T17:57:16.328 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-05-04T17:57:16.328 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (1 ms) 2024-05-04T17:57:23.828 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:18.759 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:23.832 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:18.763 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:23.878 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:18.808 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:23.882 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:18.771 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:23.886 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:18.775 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:23.894 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:18.825 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:23.897 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:18.828 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:23.927 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:18.816 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:23.942 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:18.873 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:25.050 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:19.938 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:25.050 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:19.940 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:25.061 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:19.951 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:25.245 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:20.175 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:25.245 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:20.176 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:25.257 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:20.188 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:25.580 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:20.511 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:25.581 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:20.512 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:25.615 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:20.546 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:30.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [-------Aio.RoundTripAppendPP (3457 ms) 2024-05-04T17:57:30.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-05-04T17:57:30.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3081 ms) 2024-05-04T17:57:30.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-05-04T17:57:30.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (4418 ms) 2024-05-04T17:57:30.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-05-04T17:57:30.156 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (4765 ms) 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: setting pool EIO 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: max_success 0, min_failed 1 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4544 ms) 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4855 ms) 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (532707 ms total) 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3288 ms) 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-05-04T17:57:30.157 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (10024 ms) 2024-05-04T17:57:30.158 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-05-04T17:57:30.158 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (16032 ms) 2024-05-04T17:57:30.158 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-05-04T17:57:30.158 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3092 ms) 2024-05-04T17:57:30.158 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (32436 ms total) 2024-05-04T17:57:30.158 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: 2024-05-04T17:57:30.158 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-05-04T17:57:30.158 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-05-04T17:57:30.158 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3668 ms) 2024-05-04T17:57:30.158 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3668 ms total) 2024-05-04T17:57:30.158 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: 2024-05-04T17:57:30.158 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14380 ms) 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7037 ms) 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7869 ms) 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7340 ms) 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3468 ms) 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7825 ms) 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-05-04T17:57:30.159 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7026 ms) 2024-05-04T17:57:30.160 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-05-04T17:57:30.160 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7933 ms) 2024-05-04T17:57:30.160 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-05-04T17:57:30.160 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7585 ms) 2024-05-04T17:57:30.160 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-05-04T17:57:30.160 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7298 ms) 2024-05-04T17:57:30.160 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-05-04T17:57:30.160 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (8227 ms) 2024-05-04T17:57:30.160 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-05-04T17:57:30.160 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6820 ms) 2024-05-04T17:57:30.160 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-05-04T17:57:30.160 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7004 ms) 2024-05-04T17:57:43.942 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:38.872 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:43.998 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:38.887 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:44.001 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:38.932 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:44.003 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:38.892 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:44.056 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:38.986 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:44.056 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:38.987 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:44.058 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:38.947 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:44.060 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:38.991 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:44.113 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:39.044 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:44.648 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2---] 22 tests from LibRadosMiscPP (646886 ms total) 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (2241 ms) 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (2241 ms total) 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (367 ms) 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (130 ms) 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (497 ms total) 2024-05-04T17:57:44.649 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (632 ms) 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (148 ms) 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (780 ms total) 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (650 ms) 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (45 ms) 2024-05-04T17:57:44.650 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (697 ms total) 2024-05-04T17:57:44.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: 2024-05-04T17:57:44.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-05-04T17:57:44.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-05-04T17:57:44.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1683 ms) 2024-05-04T17:57:44.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1683 ms total) 2024-05-04T17:57:44.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: 2024-05-04T17:57:44.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-05-04T17:57:44.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (682967 ms total) 2024-05-04T17:57:44.651 INFO:tasks.workunit.client.0.smithi019.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-05-04T17:57:45.211 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:40.100 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:45.212 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:40.101 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:45.221 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:40.110 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:45.410 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:40.340 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:45.410 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:40.341 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:45.420 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:40.351 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:45.565 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:40.496 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:45.566 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:40.497 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:45.581 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:40.512 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:59.266 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:54.196 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:59.271 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:54.201 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:59.338 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:54.227 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:59.342 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:54.232 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:59.351 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:54.282 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:59.356 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:54.287 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:59.377 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:54.308 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:59.390 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:54.279 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:57:59.419 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:54.350 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:04.162 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:59.093 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:04.224 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:59.113 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:04.225 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:59.156 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:04.227 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:59.117 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:04.245 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:13:59.134 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:04.253 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:13:59.184 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:04.270 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:59.201 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:04.277 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:59.208 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:04.298 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:13:59.229 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:17.386 INFO:tasks.workunit.client.0.smithi019.stdout: 2024-05-04T17:58:17.386 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3029 ms) 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6980 ms) 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7576 ms) 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6892 ms) 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7907 ms) 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7230 ms) 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7614 ms) 2024-05-04T17:58:17.387 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (147040 ms total) 2024-05-04T17:58:17.388 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: 2024-05-04T17:58:17.388 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-05-04T17:58:17.388 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (715851 ms total) 2024-05-04T17:58:17.388 INFO:tasks.workunit.client.0.smithi019.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-05-04T17:58:24.371 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:19.301 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:24.373 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:19.304 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:24.418 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:19.348 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:24.430 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:19.319 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:24.431 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:19.321 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:24.434 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:19.365 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:24.438 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:19.369 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:24.475 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:19.364 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:24.483 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:19.413 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:29.464 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:24.394 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:29.469 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:24.400 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:29.476 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:24.365 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:29.478 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:24.367 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:29.496 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:24.385 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:29.501 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:24.431 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:29.522 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:24.453 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:29.529 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:24.459 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:29.545 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:24.476 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:44.336 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:39.266 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:44.337 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:39.268 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:44.354 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:39.285 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:44.386 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:39.276 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:44.388 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:39.277 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:44.393 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:39.324 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:44.396 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:39.326 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:44.406 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:39.296 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:44.420 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:39.351 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:49.348 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:44.278 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:49.351 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:44.282 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:49.364 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:44.294 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:49.400 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:44.289 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:49.401 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:44.290 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:49.402 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:44.333 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:49.405 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:44.336 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:49.413 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:44.302 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:58:49.420 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:44.351 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:04.618 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:59.548 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:04.680 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:59.611 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:04.697 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:59.586 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:04.710 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:14:59.641 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:04.716 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:59.606 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:04.746 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:59.677 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:04.750 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:59.681 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:04.772 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:14:59.661 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:04.790 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:14:59.721 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:19.545 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:14.475 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:19.549 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:14.479 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:19.576 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:14.507 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:19.589 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:14.520 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:19.593 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:14.524 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:19.615 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:14.504 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:19.616 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:14.505 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:19.624 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:14.555 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:19.647 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:14.536 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:24.647 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:19.577 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:24.705 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:19.635 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:24.706 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:19.595 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:24.728 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:19.659 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:24.737 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:19.626 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:24.781 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:19.670 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:24.784 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:19.715 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:24.785 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:19.716 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:24.812 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:19.742 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:34.596 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:29.527 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:34.598 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:29.529 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:34.628 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:29.558 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:34.653 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:29.542 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:34.656 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:29.545 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:34.656 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:29.587 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:34.660 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:29.591 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:34.690 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:29.620 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:34.716 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:29.605 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:44.739 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:39.669 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:44.794 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:39.725 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:44.819 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:39.749 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:44.855 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:39.744 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:44.856 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:39.745 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:44.869 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:39.800 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:44.872 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:39.803 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:44.880 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:39.769 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:44.900 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:39.831 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:49.745 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:44.675 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:49.750 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:44.681 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:49.756 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:44.687 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:49.757 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:44.646 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:49.809 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:44.699 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:49.852 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:44.750 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:49.852 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:44.722 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:49.852 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:44.763 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:49.852 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:44.776 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T17:59:59.599 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-05-04T17:59:59.599 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-05-04T17:59:59.599 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-05-04T17:59:59.599 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: seed 38691 2024-05-04T17:59:59.599 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-05-04T17:59:59.599 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (30180 ms) 2024-05-04T17:59:59.599 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-05-04T17:59:59.599 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (121625 ms) 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (11269 ms) 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (163075 ms total) 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (107192 ms) 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (116987 ms) 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (52662 ms) 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-05-04T17:59:59.600 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: my_snaps [3] 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: my_snaps [4,3] 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: my_snaps [5,4,3] 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: promoting some heads 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: promoting from clones for snap 6 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: promoting from clones for snap 5 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: promoting from clones for snap 4 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: promoting from clones for snap 3 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: waiting for scrubs... 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: done waiting 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (191105 ms) 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (12199 ms) 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-05-04T17:59:59.601 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (10178 ms) 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8920 ms) 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10287 ms) 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (12442 ms) 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (10302 ms) 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (11230 ms) 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (42568 ms) 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10770 ms) 2024-05-04T17:59:59.602 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-05-04T17:59:59.603 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (10169 ms) 2024-05-04T17:59:59.603 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-05-04T17:59:59.603 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (15569 ms) 2024-05-04T17:59:59.603 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-05-04T17:59:59.603 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (9827 ms) 2024-05-04T17:59:59.603 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-05-04T17:59:59.603 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (13256 ms) 2024-05-04T18:00:04.730 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:59.660 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:04.731 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:59.662 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:04.761 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:15:59.691 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:04.783 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:59.713 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:04.784 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:59.716 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:04.812 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:59.701 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:04.813 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:15:59.744 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:04.814 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:59.703 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:04.840 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:15:59.730 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:05.859 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:00.748 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:05.860 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:00.749 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:05.868 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:00.758 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:05.919 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:00.850 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:05.920 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:00.851 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:05.929 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:00.860 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:06.013 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:00.943 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:06.014 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:00.945 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:06.023 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:00.954 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:18.966 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:13.896 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:18.974 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:13.904 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:18.974 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:13.863 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:18.975 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:13.865 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:19.005 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:13.894 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:19.008 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:13.939 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:19.034 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:13.965 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:19.035 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:13.966 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:19.082 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:14.013 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:19.886 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:14.775 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:19.896 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:14.785 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:19.973 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:14.862 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:20.070 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:14.998 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:20.070 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:14.999 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:20.078 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:15.009 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:20.176 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:15.106 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:20.177 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:15.107 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:20.188 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:15.118 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:22.537 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-05-04T18:00:22.537 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: hmm, no HitSet yet 2024-05-04T18:00:22.537 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: ok, hit_set contains 273:602f83fe:::foo:head 2024-05-04T18:00:22.537 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (10347 ms) 2024-05-04T18:00:22.537 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-05-04T18:00:22.537 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg_num = 32 2024-05-04T18:00:22.537 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 0 ls 1714845617,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 1 ls 1714845617,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 2 ls 1714845616,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 3 ls 1714845616,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 4 ls 1714845617,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 5 ls 1714845616,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 6 ls 1714845617,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 7 ls 1714845617,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 8 ls 1714845617,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 9 ls 1714845617,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 10 ls 1714845617,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 11 ls 1714845617,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 12 ls 1714845616,0 2024-05-04T18:00:22.538 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 13 ls 1714845617,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 14 ls 1714845616,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 15 ls 1714845616,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 16 ls 1714845616,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 17 ls 1714845617,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 18 ls 1714845617,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 19 ls 1714845617,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 20 ls 1714845616,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 21 ls 1714845617,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 22 ls 1714845616,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 23 ls 1714845616,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 24 ls 1714845617,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 25 ls 1714845616,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 26 ls 1714845617,0 2024-05-04T18:00:22.539 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 27 ls 1714845616,0 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 28 ls 1714845616,0 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 29 ls 1714845616,0 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 30 ls 1714845617,0 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg 31 ls 1714845616,0 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: pg_num = 32 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:6cac518f:::0:head 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:02547ec2:::1:head 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:f905c69b:::2:head 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:cfc208b3:::3:head 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:d83876eb:::4:head 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:b29083e3:::5:head 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:c4fdafeb:::6:head 2024-05-04T18:00:22.540 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:5c6b0b28:::7:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:bd63b0f1:::8:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:e960b815:::9:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:52ea6a34:::10:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:89d3ae78:::11:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:de5d7c5f:::12:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:566253c9:::13:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:62a1935d:::14:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:863748b0:::15:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:3958e169:::16:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:4d4dabf9:::17:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:8391935d:::18:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:28883081:::19:head 2024-05-04T18:00:22.541 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:69259c59:::20:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:4bdb80b7:::21:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:a11c5d71:::22:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:271af37b:::23:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:95b121be:::24:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:58d1031b:::25:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:0a050783:::26:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:c709704c:::27:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:cbe56eaf:::28:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:86b4b162:::29:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:70d89383:::30:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:dd450c7c:::31:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:6d5729b1:::32:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:c388f3fb:::33:head 2024-05-04T18:00:22.542 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:56cfea31:::34:head 2024-05-04T18:00:22.543 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:9dbc1bf7:::35:head 2024-05-04T18:00:22.543 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:40b74ccd:::36:head 2024-05-04T18:00:22.543 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:4d5aaf42:::37:head 2024-05-04T18:00:22.543 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:920f362c:::38:head 2024-05-04T18:00:22.543 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:6cc53222:::39:head 2024-05-04T18:00:22.544 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:9cad833f:::40:head 2024-05-04T18:00:22.544 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:1ea84d41:::41:head 2024-05-04T18:00:22.544 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:c4480ef6:::42:head 2024-05-04T18:00:22.544 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:a694361e:::43:head 2024-05-04T18:00:22.544 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:d1bd33e9:::44:head 2024-05-04T18:00:22.544 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:ddc2cd5d:::45:head 2024-05-04T18:00:22.544 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:2b782207:::46:head 2024-05-04T18:00:22.544 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:7b187fca:::47:head 2024-05-04T18:00:22.545 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:90ecdf6f:::48:head 2024-05-04T18:00:22.545 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:a5ed95fe:::49:head 2024-05-04T18:00:22.545 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:ea0eaa55:::50:head 2024-05-04T18:00:22.545 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:f33ef17b:::51:head 2024-05-04T18:00:22.545 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:a0d1b2f6:::52:head 2024-05-04T18:00:22.545 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:60c5229e:::53:head 2024-05-04T18:00:22.545 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:edcbc575:::54:head 2024-05-04T18:00:22.545 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:102cf253:::55:head 2024-05-04T18:00:22.545 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:efb7fb0b:::56:head 2024-05-04T18:00:22.545 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:50d0a326:::57:head 2024-05-04T18:00:22.545 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:d4dc5daf:::58:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:3a130462:::59:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:ec87ed71:::60:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:d5bc9454:::61:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:3ddfe313:::62:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:7c2816b9:::63:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:47e00e4d:::64:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:c6410c18:::65:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:b48ed237:::66:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:cd63ad31:::67:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:b179e92b:::68:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:0d9f741a:::69:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:6d3352ae:::70:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:c6d5c19e:::71:head 2024-05-04T18:00:22.546 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:bc4729c3:::72:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:77e930b9:::73:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:0abeecfd:::74:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:b7c37e15:::75:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:b6378398:::76:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:02bd68de:::77:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:cc795d2d:::78:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:630d4fea:::79:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:e0d29ef5:::80:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:fd6f13d2:::81:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:606461d5:::82:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:eadbdc43:::83:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:8761d0bb:::84:head 2024-05-04T18:00:22.547 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:9ef0186f:::85:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:e0d41294:::86:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:961de695:::87:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:1423148f:::88:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:633a8fa2:::89:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:a8653809:::90:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:3dac8b33:::91:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:35aad435:::92:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:f6dcc343:::93:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:dbbdad87:::94:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:1cb48ce0:::95:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:03cd461c:::96:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:17a4ea99:::97:head 2024-05-04T18:00:22.548 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:9993c9a7:::98:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:6394211c:::99:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:94c7ae57:::100:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:6fdee5bb:::101:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:9a477fd1:::102:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:eb850916:::103:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:affc56b9:::104:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:b42dc814:::105:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:f319f8f0:::106:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:9a40b9de:::107:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:8b524f28:::108:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:e3de589f:::109:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:90f90a5b:::110:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:a7b4f1d7:::111:head 2024-05-04T18:00:22.549 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:af51766e:::112:head 2024-05-04T18:00:22.550 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:b6f90bd1:::113:head 2024-05-04T18:00:22.550 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:e0261208:::114:head 2024-05-04T18:00:22.550 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:c9569ef7:::115:head 2024-05-04T18:00:22.550 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:61bebe50:::116:head 2024-05-04T18:00:22.550 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:fe93412b:::117:head 2024-05-04T18:00:22.550 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:d3d38bee:::118:head 2024-05-04T18:00:22.550 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:3100ba0c:::119:head 2024-05-04T18:00:22.550 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:d0560ada:::120:head 2024-05-04T18:00:22.550 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:f0ea8b35:::121:head 2024-05-04T18:00:22.550 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:766f231a:::122:head 2024-05-04T18:00:22.550 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:a07a2582:::123:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:bd7c6b3a:::124:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:fb2ddaff:::125:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:4408e1fe:::126:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:ee1df7a7:::127:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:c3002909:::128:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:4f48ffa9:::129:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:edf38733:::130:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:c08425c0:::131:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:5f902d98:::132:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:41ea2c93:::133:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:813cee13:::134:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:0131818d:::135:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:26ba5a85:::136:head 2024-05-04T18:00:22.551 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:381b8a5a:::137:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:28797e47:::138:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:bfca7f22:::139:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:36807075:::140:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:80b03975:::141:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:5c15709b:::142:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:f39ea15e:::143:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:ea992956:::144:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:48887b1c:::145:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:9f24a9dd:::146:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:987f100b:::147:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:d2dd3581:::148:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:7fed1808:::149:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:c80b70e9:::150:head 2024-05-04T18:00:22.552 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:85ed90f9:::151:head 2024-05-04T18:00:22.553 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:36428b24:::152:head 2024-05-04T18:00:22.553 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:d044c34a:::153:head 2024-05-04T18:00:22.553 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:7c18bf58:::154:head 2024-05-04T18:00:22.553 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:d1c21232:::155:head 2024-05-04T18:00:22.553 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:a7a3c575:::156:head 2024-05-04T18:00:22.553 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:87da0633:::157:head 2024-05-04T18:00:22.553 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:d5ac3822:::158:head 2024-05-04T18:00:22.553 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:3f20522d:::159:head 2024-05-04T18:00:22.553 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:6ca26563:::160:head 2024-05-04T18:00:22.553 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:532ce135:::161:head 2024-05-04T18:00:22.553 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:c78863e6:::162:head 2024-05-04T18:00:25.055 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:19.985 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:25.057 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:19.988 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:25.071 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:20.002 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:25.107 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:20.037 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:25.108 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:19.997 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:25.111 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:20.042 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:25.113 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:20.002 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:25.125 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:20.056 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:25.126 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:20.016 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:35.026 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:29.956 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:35.028 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:29.958 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:35.057 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:29.988 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:35.079 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:29.968 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:35.080 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:29.969 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:35.099 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:30.030 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:35.101 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:30.033 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:35.103 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:29.992 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:35.126 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:30.057 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:44.950 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:39.880 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:44.951 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:39.882 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:44.970 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:39.901 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:44.998 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:39.887 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:44.999 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:39.888 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:45.002 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:39.933 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:45.004 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:39.935 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:45.016 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:39.906 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:45.023 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:39.953 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:55.170 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:50.101 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:55.176 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:50.107 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:55.214 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:50.145 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:55.242 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:50.132 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:55.245 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:50.134 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:55.245 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:50.176 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:55.249 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:50.180 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:55.278 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:50.209 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:55.281 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:50.170 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:56.165 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:51.054 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:56.166 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:51.055 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:56.176 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:16:51.066 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:56.288 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:51.219 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:56.290 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:51.221 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:56.305 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:16:51.236 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:56.347 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:51.278 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:56.348 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:51.279 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:00:56.358 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:16:51.288 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:05.229 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:00.159 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:05.233 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:00.164 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:05.258 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:00.189 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:05.282 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:00.171 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:05.287 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:00.218 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:05.288 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:00.176 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:05.290 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:00.221 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:05.312 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:00.201 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:05.327 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:00.258 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:10.256 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:05.187 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:10.261 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:05.192 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:10.276 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:05.207 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:10.319 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:05.208 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:10.320 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:05.209 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:10.322 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:05.253 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:10.323 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:05.254 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:10.338 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:05.227 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:10.339 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:05.269 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:25.303 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:20.233 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:25.304 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:20.235 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:25.328 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:20.259 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:25.356 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:20.245 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:25.357 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:20.247 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:25.359 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:20.290 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:25.361 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:20.292 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:25.379 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:20.268 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:25.384 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:20.315 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:26.079 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:20.968 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:26.080 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:20.969 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:26.088 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:20.978 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:26.149 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:21.080 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:26.150 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:21.081 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:26.151 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:21.082 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:26.152 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:21.084 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:26.160 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:21.091 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:26.162 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:21.093 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:33.867 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:7462ddf6:::163:head 2024-05-04T18:01:33.868 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:5d165639:::164:head 2024-05-04T18:01:33.868 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:f43765fc:::165:head 2024-05-04T18:01:33.868 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:b4c720e9:::166:head 2024-05-04T18:01:33.868 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:e694b040:::167:head 2024-05-04T18:01:33.868 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:afa38db2:::168:head 2024-05-04T18:01:33.868 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:77ba9f53:::169:head 2024-05-04T18:01:33.868 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:87495034:::170:head 2024-05-04T18:01:33.868 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:7c96bf0e:::171:head 2024-05-04T18:01:33.868 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:dbe346cc:::172:head 2024-05-04T18:01:33.869 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:e943ec24:::173:head 2024-05-04T18:01:33.869 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:f97a9c0c:::174:head 2024-05-04T18:01:33.869 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:6f26e74d:::175:head 2024-05-04T18:01:33.869 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:4f95e106:::176:head 2024-05-04T18:01:33.869 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:0e6f2f8f:::177:head 2024-05-04T18:01:33.869 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:05db05f1:::178:head 2024-05-04T18:01:33.869 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:38a78d66:::179:head 2024-05-04T18:01:33.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:d095610b:::180:head 2024-05-04T18:01:33.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:a1a9d709:::181:head 2024-05-04T18:01:33.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:1e5d39db:::182:head 2024-05-04T18:01:33.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:f7df4fb9:::183:head 2024-05-04T18:01:33.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:03a7f161:::184:head 2024-05-04T18:01:33.870 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:ba70721e:::185:head 2024-05-04T18:01:33.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:28e5662d:::186:head 2024-05-04T18:01:33.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:973d52de:::187:head 2024-05-04T18:01:33.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:4303eb1c:::188:head 2024-05-04T18:01:33.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:b990b48e:::189:head 2024-05-04T18:01:33.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:29b8165b:::190:head 2024-05-04T18:01:33.871 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:3547f197:::191:head 2024-05-04T18:01:33.872 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:7e260936:::192:head 2024-05-04T18:01:33.872 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:1abec7b1:::193:head 2024-05-04T18:01:33.872 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:10fdda93:::194:head 2024-05-04T18:01:33.872 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:15817eea:::195:head 2024-05-04T18:01:33.872 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:770bab57:::196:head 2024-05-04T18:01:33.872 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:ed9e13e7:::197:head 2024-05-04T18:01:33.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:71471a8f:::198:head 2024-05-04T18:01:33.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: checking for 274:10fb1d02:::199:head 2024-05-04T18:01:33.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (22001 ms) 2024-05-04T18:01:33.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-05-04T18:01:33.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845640,0 2024-05-04T18:01:33.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: first is 1714845640 2024-05-04T18:01:33.873 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845640,0 2024-05-04T18:01:33.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845640,0 2024-05-04T18:01:33.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845640,1714845642,1714845643,0 2024-05-04T18:01:33.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845640,1714845642,1714845643,0 2024-05-04T18:01:33.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845640,1714845642,1714845643,0 2024-05-04T18:01:33.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845640,1714845642,1714845643,1714845645,1714845646,0 2024-05-04T18:01:33.874 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845640,1714845642,1714845643,1714845645,1714845646,0 2024-05-04T18:01:33.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845640,1714845642,1714845643,1714845645,1714845646,0 2024-05-04T18:01:33.875 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845640,1714845642,1714845643,1714845645,1714845646,1714845649,1714845650,0 2024-05-04T18:01:33.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845640,1714845642,1714845643,1714845645,1714845646,1714845649,1714845650,0 2024-05-04T18:01:33.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845640,1714845642,1714845643,1714845645,1714845646,1714845649,1714845650,0 2024-05-04T18:01:33.877 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714845643,1714845645,1714845646,1714845649,1714845650,1714845652,1714845653,0 2024-05-04T18:01:33.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: first now 1714845643, trimmed 2024-05-04T18:01:33.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22438 ms) 2024-05-04T18:01:33.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-05-04T18:01:33.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: foo0 2024-05-04T18:01:33.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-05-04T18:01:33.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (17850 ms) 2024-05-04T18:01:33.878 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-05-04T18:01:40.333 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:35.263 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:40.337 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:35.268 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:40.366 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:35.296 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:40.390 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:35.280 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:40.391 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:35.281 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:40.403 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:35.334 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:40.404 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:35.335 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:40.417 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:35.306 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:40.433 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:35.364 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:45.377 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:40.307 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:45.382 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:40.310 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:45.394 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:40.325 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:45.427 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:40.316 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:45.428 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:40.359 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:45.428 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:40.318 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:45.429 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:40.360 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:45.442 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:40.332 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:45.450 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:40.381 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:55.418 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:50.349 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:55.420 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:50.351 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:55.438 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:17:50.369 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:55.471 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:50.360 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:55.472 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:50.361 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:55.473 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:50.404 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:55.477 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:50.408 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:55.488 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:17:50.377 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:01:55.492 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:17:50.423 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:05.481 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:00.412 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:05.484 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:00.414 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:05.513 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:00.444 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:05.537 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:00.427 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:05.541 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:00.430 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:05.556 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:00.487 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:05.560 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:00.491 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:05.566 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:00.456 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:05.598 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:00.529 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:10.648 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:05.579 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:10.652 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:05.582 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:10.673 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:05.603 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:10.716 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:05.605 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:10.735 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:05.666 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:10.736 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:05.625 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:10.737 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:05.668 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:10.755 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:05.644 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:10.761 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:05.692 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:25.749 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:20.679 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:25.752 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:20.683 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:25.787 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:20.718 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:25.806 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:20.696 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:25.810 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:20.699 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:25.813 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:20.744 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:25.815 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:20.746 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:25.846 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:20.735 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:25.851 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:20.782 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:40.774 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:35.704 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:40.775 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:35.706 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:40.801 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:35.732 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:40.816 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:35.747 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:40.824 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:35.713 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:40.832 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:35.721 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:40.833 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:35.764 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:40.856 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:35.745 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:40.867 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:35.797 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:41.723 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:36.612 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:41.724 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:36.613 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:41.736 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:36.625 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:41.812 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:36.743 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:41.813 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:36.744 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:41.826 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:36.757 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:41.829 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:36.760 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:41.830 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:36.761 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:41.842 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:36.773 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:45.654 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:40.584 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:45.658 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:40.589 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:45.670 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:40.601 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:45.705 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:40.594 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:45.707 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:40.638 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:45.708 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:40.639 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:45.709 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:40.598 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:45.720 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:40.610 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:02:45.721 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:40.652 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:00.849 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:55.779 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:00.853 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:55.784 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:00.872 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:18:55.802 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:00.903 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:55.792 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:00.904 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:55.794 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:00.907 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:55.838 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:00.912 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:55.842 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:00.922 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:18:55.811 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:00.930 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:18:55.861 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:05.875 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:00.806 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:05.953 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:00.883 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:06.003 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:00.934 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:06.007 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:00.896 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:06.012 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:00.901 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:06.032 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:00.921 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:06.043 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:00.974 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:06.047 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:00.978 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:06.070 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:01.001 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:11.074 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:06.005 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:11.134 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:06.065 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:11.137 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:06.068 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:11.151 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:06.082 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:11.159 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:06.090 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:11.176 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:06.107 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:11.197 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:06.087 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:11.200 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:06.089 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:11.218 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:06.107 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:16.112 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:11.042 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:16.178 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:11.109 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:16.195 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:11.125 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:16.222 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:11.112 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:16.226 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:11.157 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:16.226 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:11.116 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:16.232 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:11.163 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:16.241 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:11.131 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:16.250 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:11.181 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:31.225 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:26.155 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:31.230 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:26.161 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:31.232 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:26.163 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:31.275 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:26.206 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:31.301 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:26.190 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:31.309 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:26.199 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:31.311 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:26.243 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:31.343 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:26.233 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:31.363 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:26.294 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:46.078 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:41.008 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:46.133 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:41.063 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:46.162 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:41.092 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:46.186 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:41.075 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:46.188 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:41.077 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:46.199 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:41.130 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:46.201 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:41.132 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:46.214 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:41.102 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:46.225 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:41.156 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:51.149 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:46.079 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:51.153 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:46.084 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:51.170 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:19:46.101 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:51.199 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:46.089 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:51.201 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:46.090 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:51.206 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:46.137 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:51.211 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:46.142 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:51.211 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:19:46.101 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:03:51.226 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:19:46.157 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:11.315 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:06.246 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:11.319 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:06.250 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:11.350 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:06.281 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:11.368 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:06.258 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:11.372 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:06.261 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:11.384 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:06.315 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:11.388 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:06.319 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:11.396 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:06.285 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:11.417 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:06.347 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:26.595 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:21.525 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:26.597 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:21.528 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:26.634 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:21.565 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:26.658 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:21.547 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:26.660 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:21.550 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:26.678 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:21.609 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:26.684 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:21.615 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:26.698 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:21.587 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:26.727 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:21.658 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:31.575 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:26.506 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:31.581 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:26.512 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:31.628 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:26.558 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:31.634 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:26.565 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:31.654 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:26.543 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:31.662 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:26.593 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:31.681 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:26.570 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:31.683 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:26.615 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:31.698 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:26.587 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:41.636 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:36.567 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:41.638 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:36.569 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:41.661 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:36.589 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:41.687 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:36.576 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:41.688 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:36.578 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:41.689 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:36.620 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:41.690 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:36.621 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:41.709 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:36.599 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:41.713 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:36.644 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:51.615 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:46.546 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:51.617 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:46.548 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:51.629 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:46.560 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:51.664 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:46.553 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:51.666 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:46.555 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:51.677 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:46.566 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:51.688 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:46.619 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:51.689 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:46.620 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:51.701 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:46.632 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:04:56.301 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T18:04:56.286+0000 2628e640 -1 snap_mapper.add_oid found existing snaps mapped on 292:e72c28c0:test-rados-api-smithi019-38691-80::foo:2, removing 2024-05-04T18:04:56.318 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T18:04:56.309+0000 302a2640 -1 snap_mapper.add_oid found existing snaps mapped on 292:e72c28c0:test-rados-api-smithi019-38691-80::foo:2, removing 2024-05-04T18:05:01.808 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:56.738 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:01.813 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:56.744 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:01.841 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:20:56.771 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:01.872 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:56.761 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:01.876 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:56.765 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:01.892 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:56.823 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:01.894 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:56.825 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:01.897 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:20:56.786 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:01.920 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:20:56.851 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:11.676 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:06.607 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:11.678 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:06.609 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:11.701 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:06.632 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:11.724 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:06.614 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:11.725 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:06.615 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:11.729 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:06.661 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:11.733 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:06.662 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:11.748 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:06.637 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:11.755 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:06.686 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:14.666 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:09.596 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:14.673 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:09.604 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:14.680 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:09.570 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:14.690 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:09.621 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:14.692 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:09.623 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:14.734 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:09.624 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:14.754 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:09.644 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:14.764 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:09.694 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:14.776 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:09.707 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:16.709 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:11.639 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:16.711 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:11.642 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:16.724 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:11.655 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:16.771 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:11.660 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:16.787 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:11.718 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:16.789 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:11.720 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:16.797 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:11.687 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:16.800 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:11.731 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:16.808 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:11.697 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:30.266 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (21632 ms) 2024-05-04T18:05:30.266 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (26183 ms) 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3638 ms) 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (5946 ms) 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4499 ms) 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (5331 ms) 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5579 ms) 2024-05-04T18:05:30.267 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39981 ms) 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (18272 ms) 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5708 ms) 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4710 ms) 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.268 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (27981 ms) 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16987 ms) 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6940 ms) 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5421 ms) 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: waiting for scrubs... 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: done waiting 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26044 ms) 2024-05-04T18:05:30.269 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-05-04T18:05:30.270 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.270 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (11192 ms) 2024-05-04T18:05:30.270 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-05-04T18:05:30.270 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.270 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (11070 ms) 2024-05-04T18:05:30.270 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-05-04T18:05:30.270 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:05:30.270 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (10917 ms) 2024-05-04T18:05:31.820 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:26.750 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:31.821 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:26.752 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:31.854 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:26.785 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:31.873 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:26.763 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:31.878 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:26.768 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:31.886 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:26.817 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:31.891 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:26.822 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:31.907 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:26.797 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:31.922 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:26.853 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:32.753 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:27.642 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:32.754 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:27.644 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:32.765 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:27.654 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:33.009 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:27.940 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:33.010 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:27.941 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:33.021 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:27.952 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:33.202 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:28.132 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:33.203 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:28.134 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:33.215 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:28.146 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:47.021 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:41.952 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:47.024 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:41.955 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:47.057 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:41.987 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:47.077 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:41.966 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:47.078 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:41.968 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:47.081 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:42.012 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:47.082 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:42.013 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:47.111 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:42.000 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:47.131 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:42.062 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:52.043 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:46.973 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:52.045 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:46.976 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:52.059 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:46.989 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:52.090 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:46.980 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:52.091 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:46.981 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:52.094 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:47.025 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:52.096 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:47.027 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:52.105 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:46.994 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:05:52.110 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:47.041 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:02.057 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:56.987 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:02.060 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:56.991 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:02.083 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:21:57.014 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:02.111 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:57.000 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:02.112 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:57.001 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:02.112 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:57.043 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:02.114 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:57.045 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:02.133 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:21:57.023 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:02.137 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:21:57.068 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:11.975 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:06.905 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:11.977 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:06.908 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:11.994 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:06.925 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:12.027 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:06.916 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:12.028 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:06.917 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:12.033 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:06.964 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:12.034 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:06.965 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:12.039 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:06.929 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:12.050 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:06.981 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:22.078 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:17.009 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:22.130 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:17.019 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:22.131 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:17.021 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:22.145 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:17.076 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:22.156 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:17.045 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:22.164 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:17.095 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:22.209 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:17.140 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:22.210 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:17.142 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:22.226 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:17.157 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:32.204 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:27.134 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:32.207 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:27.138 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:32.237 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:27.168 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:32.262 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:27.152 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:32.265 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:27.155 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:32.282 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:27.213 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:32.283 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:27.214 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:32.285 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:27.174 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:32.313 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:27.244 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:37.074 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:32.005 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:37.077 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:32.008 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:37.087 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:32.018 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:37.126 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:32.015 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:37.127 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:32.058 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:37.127 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:32.017 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:37.128 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:32.059 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:37.137 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:32.026 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:37.138 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:32.069 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:38.365 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T18:06:38.360+0000 2628e640 -1 snap_mapper.add_oid found existing snaps mapped on 95:d7983570:test-rados-api-smithi019-38691-97::foo:21, removing 2024-05-04T18:06:38.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T18:06:38.388+0000 2dea2640 -1 snap_mapper.add_oid found existing snaps mapped on 95:d7983570:test-rados-api-smithi019-38691-97::foo:21, removing 2024-05-04T18:06:52.284 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:47.215 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:52.291 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:47.222 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:52.335 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:47.266 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:52.341 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:22:47.272 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:52.347 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:47.236 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:52.351 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:47.240 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:52.370 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:47.301 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:52.389 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:22:47.279 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:06:52.409 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:22:47.340 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:07.285 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:02.216 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:07.292 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:02.223 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:07.294 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:02.183 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:07.296 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:02.185 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:07.297 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:02.228 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:07.300 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:02.231 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:07.378 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:02.267 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:07.391 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:02.322 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:07.392 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:02.323 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:17.244 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:12.175 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:17.248 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:12.179 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:17.273 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:12.204 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:17.299 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:12.188 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:17.301 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:12.191 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:17.302 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:12.233 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:17.303 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:12.235 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:17.322 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:12.212 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:17.327 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:12.258 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:27.456 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:22.387 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:27.461 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:22.392 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:27.489 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:22.420 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:27.516 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:22.405 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:27.519 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:22.409 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:27.527 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:22.459 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:27.530 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:22.461 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:27.551 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:22.441 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:27.556 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:22.487 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:29.790 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:24.721 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:29.793 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:24.724 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:29.812 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:24.743 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:29.842 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:24.731 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:29.847 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:24.736 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:29.848 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:24.779 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:29.849 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:24.780 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:29.862 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:24.752 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:29.867 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:24.798 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:37.342 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:32.273 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:37.343 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:32.274 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:37.364 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:32.295 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:37.396 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:32.327 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:37.396 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:32.286 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:37.399 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:32.330 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:37.402 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:32.291 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:37.421 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:32.352 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:37.422 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:32.312 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:42.498 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:37.429 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:42.502 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:37.433 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:42.521 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:37.452 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:42.556 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:37.487 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:42.558 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:37.489 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:42.558 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:37.448 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:42.560 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:37.449 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:42.575 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:37.506 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:42.578 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:37.467 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:57.658 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:52.545 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:57.658 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:52.547 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:57.663 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:52.593 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:57.667 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:52.598 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:57.703 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:23:52.634 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:57.705 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:23:52.594 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:57.742 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:52.673 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:57.749 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:52.680 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:07:57.793 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:23:52.724 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:12.648 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:07.537 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:12.648 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:07.538 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:12.655 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:07.586 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:12.658 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:07.589 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:12.685 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:07.575 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:12.704 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:07.635 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:12.738 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:07.669 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:12.741 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:07.672 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:12.780 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:07.711 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:17.471 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:12.401 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:17.522 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:12.453 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:17.536 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:12.425 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:17.536 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:12.467 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:17.540 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:12.429 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:17.554 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:12.444 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:17.572 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:12.504 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:17.574 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:12.505 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:17.588 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:12.520 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:27.572 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:22.503 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:27.576 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:22.507 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:27.597 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:22.528 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:27.627 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:22.516 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:27.629 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:22.519 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:27.631 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:22.562 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:27.632 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:22.564 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:27.649 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:22.538 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:27.651 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:22.583 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:37.671 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:32.602 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:37.724 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:32.613 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:37.725 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:32.615 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:37.731 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:32.662 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:37.733 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:32.664 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:37.740 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:32.671 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:37.741 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:32.630 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:37.752 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:32.683 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:37.767 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:32.698 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:47.805 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:42.734 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:47.806 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:42.737 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:47.844 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:42.774 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:47.855 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:42.745 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:47.859 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:42.749 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:47.873 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:42.804 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:47.874 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:42.805 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:47.887 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:42.777 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:47.900 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:42.831 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:57.864 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:52.795 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:57.869 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:52.800 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:57.895 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:52.826 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:57.917 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:52.807 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:57.919 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:52.808 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:57.928 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:52.860 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:57.931 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:52.862 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:57.944 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:52.833 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:08:57.958 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:52.890 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:02.874 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:57.805 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:02.878 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:57.809 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:02.901 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:24:57.832 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:02.929 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:57.818 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:02.937 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:57.826 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:02.954 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:57.886 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:02.963 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:24:57.852 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:02.982 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:57.913 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:03.012 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:24:57.943 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:17.807 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:12.737 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:17.810 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:12.741 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:17.843 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:12.774 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:17.873 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:12.762 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:17.875 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:12.806 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:17.877 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:12.766 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:17.882 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:12.813 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:17.917 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:12.806 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:17.918 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:12.849 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:18.754 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:13.643 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:18.760 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:13.649 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:18.772 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:13.661 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:18.867 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:13.798 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:18.868 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:13.799 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:18.884 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:13.815 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:18.902 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:13.833 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:18.903 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:13.834 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:18.914 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:13.845 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:32.811 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:27.741 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:32.890 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:27.821 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:32.924 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:27.855 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:32.942 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:27.831 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:32.946 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:27.878 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:32.948 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:27.838 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:32.976 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:27.907 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:32.979 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:27.869 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:33.013 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:27.944 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:33.794 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:28.684 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:33.795 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:28.685 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:33.807 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:28.697 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:33.908 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:28.839 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:33.909 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:28.840 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:33.922 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:28.853 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:33.959 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:28.890 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:33.961 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:28.891 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:33.975 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:28.906 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:34.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-05-04T18:09:34.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:09:34.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (10622 ms) 2024-05-04T18:09:34.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-05-04T18:09:34.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:09:34.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (8176 ms) 2024-05-04T18:09:34.361 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-05-04T18:09:34.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:09:34.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (7145 ms) 2024-05-04T18:09:34.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-05-04T18:09:34.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:09:34.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (27964 ms) 2024-05-04T18:09:34.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-05-04T18:09:34.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:09:34.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15038 ms) 2024-05-04T18:09:34.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-05-04T18:09:34.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12767 ms) 2024-05-04T18:09:34.362 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-05-04T18:09:34.363 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-05-04T18:09:34.363 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: 00000010 69 73 |is| 2024-05-04T18:09:34.363 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: 00000012 2024-05-04T18:09:34.363 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12085 ms) 2024-05-04T18:09:34.363 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-05-04T18:09:34.363 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: require_osd_release = squid 2024-05-04T18:09:34.363 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7472 ms) 2024-05-04T18:09:34.363 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (1077600 ms total) 2024-05-04T18:09:34.363 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: 2024-05-04T18:09:34.363 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-05-04T18:09:34.363 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-05-04T18:09:34.364 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1620 ms) 2024-05-04T18:09:34.364 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-05-04T18:09:34.364 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11375 ms) 2024-05-04T18:09:34.364 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-05-04T18:09:34.364 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (21066 ms) 2024-05-04T18:09:34.364 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-05-04T18:09:34.364 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (19 ms) 2024-05-04T18:09:34.364 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (34080 ms total) 2024-05-04T18:09:34.364 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: 2024-05-04T18:09:34.364 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-05-04T18:09:34.365 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-05-04T18:09:34.365 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (8104 ms) 2024-05-04T18:09:34.365 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-05-04T18:09:34.365 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (9474 ms) 2024-05-04T18:09:34.365 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-05-04T18:09:34.365 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: waiting for scrub... 2024-05-04T18:09:34.365 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: done waiting 2024-05-04T18:09:34.365 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (28112 ms) 2024-05-04T18:09:34.365 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-05-04T18:09:34.365 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10903 ms) 2024-05-04T18:09:34.365 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-05-04T18:09:34.366 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (9084 ms) 2024-05-04T18:09:34.366 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-05-04T18:09:34.366 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (10034 ms) 2024-05-04T18:09:34.366 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-05-04T18:09:34.366 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (12358 ms) 2024-05-04T18:09:34.366 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-05-04T18:09:42.894 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:37.825 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:42.897 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:37.828 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:42.922 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:37.852 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:42.949 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:37.839 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:42.951 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:37.840 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:42.973 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:37.904 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:42.973 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:37.863 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:42.977 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:37.909 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:43.005 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:37.936 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:52.966 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:47.897 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:52.970 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:47.901 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:52.991 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:47.922 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:53.022 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:47.912 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:53.024 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:47.913 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:53.026 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:47.958 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:53.028 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:47.959 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:53.044 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:47.933 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:09:53.051 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:47.982 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:02.970 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:57.901 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:02.972 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:57.903 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:02.996 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:25:57.927 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:03.017 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:57.906 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:03.024 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:57.914 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:03.032 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:57.963 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:03.034 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:57.965 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:03.049 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:25:57.939 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:03.058 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:25:57.989 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:08.000 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:02.930 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:08.002 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:02.933 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:08.027 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:02.958 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:08.055 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:02.942 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:08.055 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:02.943 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:08.062 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:02.993 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:08.066 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:02.997 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:08.069 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:02.959 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:08.081 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:03.012 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:23.114 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:18.045 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:23.116 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:18.047 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:23.150 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:18.081 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:23.175 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:18.065 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:23.177 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:18.066 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:23.183 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:18.114 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:23.184 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:18.115 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:23.256 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:18.146 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:23.274 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:18.206 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:24.072 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:18.961 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:24.073 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:18.962 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:24.082 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:18.972 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:24.142 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:19.073 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:24.144 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:19.075 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:24.157 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:19.089 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:24.176 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:19.107 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:24.177 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:19.108 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:24.188 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:19.119 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:38.128 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:33.056 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:38.186 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:33.075 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:38.195 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:33.085 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:38.199 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:33.130 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:38.199 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:33.130 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:38.202 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:33.133 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:38.234 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:33.123 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:38.257 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:33.188 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:38.261 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:33.192 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:43.171 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:38.101 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:43.173 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:38.104 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:43.195 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:38.126 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:43.220 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:38.109 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:43.221 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:38.110 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:43.227 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:38.158 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:43.231 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:38.162 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:43.236 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:38.126 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:43.249 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:38.180 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:53.404 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:48.335 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:53.405 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:48.294 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:53.406 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:48.296 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:53.411 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:48.342 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:53.443 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:48.332 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:53.477 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:48.408 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:53.480 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:48.411 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:53.487 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:48.418 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:10:53.511 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:48.442 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:03.236 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:58.167 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:03.237 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:58.168 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:03.250 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:26:58.181 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:03.282 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:58.171 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:03.285 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:58.174 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:03.288 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:58.219 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:03.289 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:58.221 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:03.297 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:26:58.187 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:03.302 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:26:58.233 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:13.273 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:08.203 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:13.275 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:08.206 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:13.295 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:08.226 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:13.322 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:08.212 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:13.324 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:08.213 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:13.327 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:08.258 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:13.328 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:08.259 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:13.345 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:08.235 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:13.353 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:08.284 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:23.364 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:18.294 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:23.425 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:18.314 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:23.428 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:18.318 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:23.430 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:18.361 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:23.460 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:18.349 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:23.491 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:18.422 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:23.493 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:18.424 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:23.499 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:18.431 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:23.527 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:18.458 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:28.514 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:23.404 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:28.517 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:23.448 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:28.518 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:23.408 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:28.522 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:23.453 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:28.536 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:23.426 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:28.541 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:23.472 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:28.606 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:23.537 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:28.608 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:23.539 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:28.631 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:23.562 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:43.370 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:38.300 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:43.372 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:38.303 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:43.400 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:38.330 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:43.421 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:38.310 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:43.422 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:38.311 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:43.430 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:38.362 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:43.433 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:38.364 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:43.443 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:38.332 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:43.457 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:38.388 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:47.187 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:42.118 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:47.191 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:42.122 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:47.217 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:42.148 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:47.236 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:42.125 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:47.238 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:42.128 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:47.257 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:42.147 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:47.264 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:42.196 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:47.266 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:42.197 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:47.285 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:42.216 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:48.503 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:43.434 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:48.561 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:43.450 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:48.563 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:43.494 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:48.566 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:43.456 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:48.588 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:43.478 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:48.589 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:43.519 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:48.636 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:43.567 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:48.640 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:43.571 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:11:48.654 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:43.585 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:03.540 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:58.471 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:03.542 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:58.473 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:03.575 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:58.506 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:03.594 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:58.483 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:03.595 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:58.527 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:03.596 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:58.486 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:03.597 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:58.528 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:03.627 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:58.558 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:03.629 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:58.518 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:04.392 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:59.282 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:04.393 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:59.283 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:04.402 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:27:59.291 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:04.458 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:59.389 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:04.459 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:59.390 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:04.469 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:27:59.400 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:04.473 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:59.404 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:04.474 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:59.405 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:04.483 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:27:59.414 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:18.643 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:13.573 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:18.692 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:13.582 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:18.694 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:13.584 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:18.711 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:13.642 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:18.721 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:13.611 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:18.751 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:13.682 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:18.782 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:13.713 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:18.784 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:13.715 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:18.814 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:13.745 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:23.623 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:18.553 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:23.624 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:18.555 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:23.639 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:18.570 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:23.672 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:18.562 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:23.675 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:18.564 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:23.689 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:18.578 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:23.699 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:18.630 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:23.700 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:18.631 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:23.715 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:18.647 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:33.809 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:28.740 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:33.865 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:28.754 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:33.866 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:28.756 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:33.877 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:28.808 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:33.896 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:28.786 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:33.917 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:28.848 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:33.933 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:28.865 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:33.940 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:28.871 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:33.968 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:28.899 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:35.052 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (10181 ms) 2024-05-04T18:12:35.053 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-05-04T18:12:35.053 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14676 ms) 2024-05-04T18:12:35.053 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-05-04T18:12:35.053 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (10233 ms) 2024-05-04T18:12:35.053 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-05-04T18:12:35.053 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (16188 ms) 2024-05-04T18:12:35.053 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-05-04T18:12:35.053 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (10255 ms) 2024-05-04T18:12:35.053 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-05-04T18:12:35.053 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9951 ms) 2024-05-04T18:12:35.053 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-05-04T18:12:35.053 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: hmm, no HitSet yet 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: ok, hit_set contains 324:602f83fe:::foo:head 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (10309 ms) 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846254,0 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: first is 1714846254 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846254,0 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846254,0 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846254,1714846256,1714846257,0 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846254,1714846256,1714846257,0 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846254,1714846256,1714846257,0 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846254,1714846256,1714846257,1714846259,1714846260,0 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846254,1714846256,1714846257,1714846259,1714846260,0 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846254,1714846256,1714846257,1714846259,1714846260,0 2024-05-04T18:12:35.054 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846254,1714846256,1714846257,1714846259,1714846260,1714846262,1714846263,0 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846254,1714846256,1714846257,1714846259,1714846260,1714846262,1714846263,0 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846254,1714846256,1714846257,1714846259,1714846260,1714846262,1714846263,0 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: got ls 1714846257,1714846259,1714846260,1714846262,1714846263,1714846265,1714846266,0 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: first now 1714846257, trimmed 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22312 ms) 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: foo0 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (17239 ms) 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (21980 ms) 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (24889 ms) 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-05-04T18:12:35.055 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (6245 ms) 2024-05-04T18:12:35.056 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-05-04T18:12:35.056 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4415 ms) 2024-05-04T18:12:35.056 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-05-04T18:12:35.056 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (4699 ms) 2024-05-04T18:12:35.056 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-05-04T18:12:35.056 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4700 ms) 2024-05-04T18:12:35.056 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (276342 ms total) 2024-05-04T18:12:35.056 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: 2024-05-04T18:12:35.056 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-05-04T18:12:35.056 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1573370 ms total) 2024-05-04T18:12:35.056 INFO:tasks.workunit.client.0.smithi019.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-05-04T18:12:35.058 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:12:35.058 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38365 2024-05-04T18:12:35.058 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38365 2024-05-04T18:12:35.058 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:12:35.058 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38558 2024-05-04T18:12:35.058 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38558 2024-05-04T18:12:35.058 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:12:35.058 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39088 2024-05-04T18:12:35.058 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39088 2024-05-04T18:12:35.058 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38768 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38768 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38605 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38605 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38441 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38441 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38696 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38696 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39254 2024-05-04T18:12:35.059 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39254 2024-05-04T18:12:35.060 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:12:35.060 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39321 2024-05-04T18:12:35.060 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39321 2024-05-04T18:12:43.735 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:38.665 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:43.736 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:38.667 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:43.762 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:38.693 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:43.790 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:38.680 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:43.793 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:38.683 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:43.798 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:38.729 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:43.801 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:38.732 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:43.811 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:38.701 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:43.817 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:38.748 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:44.587 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:39.476 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:44.588 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:39.477 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:44.596 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:39.486 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:44.656 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:39.587 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:44.658 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:39.589 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:44.672 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:39.603 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:44.673 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:39.605 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:44.674 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:39.606 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:44.683 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:39.614 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:53.750 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:48.681 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:53.753 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:48.684 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:53.763 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:48.694 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:53.802 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:48.733 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:53.803 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:48.734 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:53.805 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:48.695 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:53.807 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:48.697 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:53.813 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:48.745 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:12:53.818 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:48.708 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:03.776 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:58.707 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:03.778 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:58.709 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:03.790 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:28:58.721 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:03.827 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:58.717 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:03.829 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:58.718 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:03.829 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:58.760 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:03.830 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:58.762 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:03.840 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:28:58.730 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:03.842 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:28:58.773 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:13.811 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:08.742 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:13.814 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:08.745 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:13.825 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:08.756 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:13.865 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:08.796 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:13.866 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:08.755 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:13.868 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:08.799 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:13.869 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:08.758 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:13.881 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:08.812 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:13.882 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:08.771 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:28.845 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:23.776 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:28.847 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:23.778 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:28.861 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:23.792 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:28.896 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:23.785 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:28.900 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:23.831 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:28.902 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:23.833 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:28.902 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:23.792 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:28.915 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:23.805 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:28.916 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:23.847 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:33.855 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:28.786 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:33.856 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:28.787 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:33.867 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:28.798 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:33.906 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:28.796 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:33.907 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:28.838 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:33.907 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:28.797 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:33.909 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:28.840 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:33.918 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:28.807 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:33.923 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:28.854 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:48.897 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:43.828 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:48.900 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:43.831 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:48.913 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:43.844 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:48.953 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:43.842 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:48.953 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:43.884 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:48.954 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:43.886 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:48.972 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:43.861 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:48.973 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:43.904 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:48.988 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:43.878 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:53.906 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:48.837 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:53.908 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:48.839 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:53.918 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:29:48.849 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:53.955 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:48.844 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:53.957 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:48.846 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:53.959 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:48.890 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:53.960 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:48.892 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:53.966 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:29:48.856 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:13:53.970 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:29:48.901 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:08.942 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:03.873 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:08.946 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:03.877 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:08.959 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:03.891 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:08.998 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:03.887 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:09.002 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:03.933 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:09.002 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:03.892 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:09.014 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:03.904 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:09.023 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:03.954 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:09.036 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:03.967 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:13.961 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:08.891 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:13.964 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:08.895 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:13.977 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:08.908 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:14.011 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:08.942 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:14.011 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:08.901 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:14.015 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:08.946 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:14.015 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:08.905 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:14.026 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:08.915 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:14.026 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:08.957 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:29.020 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:23.950 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:29.021 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:23.952 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:29.034 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:23.965 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:29.072 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:23.961 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:29.073 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:23.963 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:29.074 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:24.005 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:29.079 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:24.010 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:29.087 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:23.976 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:29.093 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:24.024 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:34.014 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:28.944 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:34.015 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:28.946 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:34.025 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:28.956 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:34.062 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:28.952 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:34.064 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:28.953 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:34.066 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:28.997 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:34.067 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:28.998 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:34.074 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:28.964 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:34.078 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:29.009 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.097 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:44.028 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.099 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:44.030 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.122 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:44.053 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.148 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:44.079 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.150 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:44.039 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.151 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:44.082 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.151 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:44.041 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.171 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:44.102 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.172 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:44.061 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.751 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:44.640 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.751 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:44.641 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.760 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:44.649 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.821 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:44.752 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.822 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:44.753 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.823 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:44.754 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.824 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:44.755 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.832 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:44.763 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:49.835 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:44.767 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:51.913 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:46.843 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:51.914 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:46.845 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:51.928 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:30:46.859 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:51.963 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:46.852 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:51.965 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:46.854 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:51.971 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:46.902 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:51.972 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:46.903 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:51.978 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:30:46.868 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:14:51.987 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:30:46.918 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:09.160 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:04.091 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:09.163 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:04.094 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:09.182 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:04.114 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:09.216 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:04.105 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:09.217 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:04.107 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:09.222 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:04.153 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:09.223 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:04.154 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:09.231 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:04.121 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:09.249 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:04.181 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:14.169 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:09.100 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:14.176 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:09.107 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:14.186 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:09.117 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:14.222 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:09.112 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:14.224 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:09.114 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:14.234 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:09.124 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:14.245 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:09.177 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:14.248 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:09.179 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:14.261 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:09.193 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:29.207 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:24.138 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:29.215 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:24.146 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:29.228 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:24.159 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:29.268 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:24.158 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:29.271 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:24.160 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:29.273 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:24.204 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:29.274 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:24.206 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:29.284 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:24.174 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:29.294 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:24.225 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:34.227 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:29.158 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:34.231 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:29.162 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:34.242 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:29.173 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:34.282 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:29.171 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:34.283 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:29.172 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:34.296 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:29.186 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:34.306 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:29.237 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:34.309 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:29.240 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:34.319 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:29.250 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:49.262 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:44.193 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:49.265 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:44.196 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:49.278 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:44.209 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:49.314 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:44.203 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:49.316 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:44.247 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:49.317 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:44.206 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:49.318 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:44.249 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:49.329 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:44.219 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:49.337 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:44.268 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:54.279 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:49.209 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:54.282 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:49.213 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:54.295 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:31:49.226 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:54.330 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:49.220 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:54.332 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:49.263 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:54.333 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:49.223 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:54.336 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:49.267 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:54.345 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:31:49.235 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:15:54.347 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:31:49.278 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:09.316 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:04.246 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:09.318 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:04.249 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:09.331 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:04.262 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:09.369 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:04.259 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:09.371 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:04.260 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:09.371 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:04.303 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:09.373 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:04.304 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:09.384 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:04.274 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:09.387 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:04.318 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:14.336 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:09.267 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:14.339 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:09.271 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:14.353 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:09.284 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:14.388 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:09.277 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:14.390 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:09.280 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:14.390 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:09.322 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:14.394 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:09.325 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:14.401 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:09.290 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:14.406 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:09.337 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:29.505 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:24.435 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:29.507 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:24.438 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:29.521 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:24.452 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:29.561 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:24.451 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:29.562 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:24.452 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:29.563 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:24.494 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:29.565 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:24.496 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:29.576 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:24.466 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:29.578 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:24.509 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:34.767 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:29.698 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:34.768 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:29.699 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:34.793 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:29.725 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:34.817 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:29.707 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:34.818 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:29.708 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:34.820 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:29.751 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:34.823 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:29.754 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:34.843 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:29.732 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:34.849 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:29.780 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:49.713 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:44.644 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:49.715 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:44.646 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:49.741 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:44.672 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:49.765 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:44.655 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:49.767 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:44.656 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:49.780 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:44.711 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:49.782 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:44.713 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:49.791 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:44.680 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:49.809 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:44.740 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:50.633 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:45.522 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:50.634 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:45.523 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:50.654 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:45.543 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:50.704 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:45.635 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:50.705 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:45.637 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:50.715 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:45.646 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:50.717 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:45.648 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:50.719 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:45.651 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:16:50.732 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:45.663 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:04.747 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:59.678 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:04.749 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:59.680 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:04.775 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:32:59.706 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:04.801 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:59.732 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:04.802 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:59.692 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:04.802 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:59.734 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:04.804 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:59.693 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:04.829 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:32:59.719 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:04.829 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:32:59.761 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:08.753 INFO:tasks.workunit.client.0.smithi019.stdout: misc: Running main() from gmock_main.cc 2024-05-04T18:17:08.754 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-05-04T18:17:08.754 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [----------] Global test environment set-up. 2024-05-04T18:17:08.754 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-05-04T18:17:08.754 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-05-04T18:17:08.754 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ OK ] NeoRadosMisc.Version (4478 ms) 2024-05-04T18:17:08.754 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-05-04T18:17:08.754 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (4863 ms) 2024-05-04T18:17:08.754 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-05-04T18:17:08.754 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ OK ] NeoRadosMisc.LongName (1807638 ms) 2024-05-04T18:17:08.754 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-05-04T18:17:08.754 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3364 ms) 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3362 ms) 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3210 ms) 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ OK ] NeoRadosMisc.Exec (3794 ms) 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3242 ms) 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3386 ms) 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3393 ms) 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-05-04T18:17:08.755 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2526 ms) 2024-05-04T18:17:08.756 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-05-04T18:17:08.756 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3457 ms) 2024-05-04T18:17:08.756 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [----------] 12 tests from NeoRadosMisc (1846713 ms total) 2024-05-04T18:17:08.756 INFO:tasks.workunit.client.0.smithi019.stdout: misc: 2024-05-04T18:17:08.756 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [----------] Global test environment tear-down 2024-05-04T18:17:08.756 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [==========] 12 tests from 1 test suite ran. (1846714 ms total) 2024-05-04T18:17:08.756 INFO:tasks.workunit.client.0.smithi019.stdout: misc: [ PASSED ] 12 tests. 2024-05-04T18:17:14.877 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:09.808 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:14.879 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:09.810 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:14.897 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:09.828 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:14.932 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:09.864 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:14.933 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:09.823 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:14.934 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:09.866 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:14.940 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:09.829 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:14.957 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:09.888 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:14.959 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:09.848 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:19.915 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:14.846 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:19.918 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:14.849 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:19.929 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:14.860 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:19.967 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:14.857 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:19.969 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:14.901 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:19.972 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:14.903 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:19.974 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:14.864 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:19.985 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:14.916 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:19.985 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:14.874 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:29.866 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:24.797 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:29.872 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:24.803 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:29.894 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:24.825 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:29.921 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:24.810 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:29.923 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:24.813 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:29.932 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:24.863 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:29.933 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:24.864 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:29.937 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:24.826 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:29.947 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:24.878 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:34.876 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:29.807 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:34.881 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:29.812 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:34.896 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:29.827 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:34.932 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:29.822 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:34.934 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:29.824 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:34.942 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:29.873 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:34.943 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:29.874 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:34.944 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:29.833 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:34.952 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:29.884 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:39.892 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:34.823 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:39.895 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:34.826 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:39.905 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:34.836 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:39.945 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:34.877 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:39.946 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:34.836 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:39.947 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:34.878 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:39.947 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:34.837 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:39.958 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:34.847 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:39.961 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:34.892 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:54.927 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:49.858 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:54.931 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:49.862 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:54.944 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:49.875 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:54.984 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:49.873 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:54.985 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:49.914 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:54.985 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:49.916 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:54.985 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:49.875 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:55.002 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:49.933 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:55.017 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:49.906 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:59.942 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:54.873 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:59.944 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:54.875 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:59.955 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:33:54.886 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:59.991 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:54.880 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:59.994 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:54.883 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:59.996 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:54.927 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:17:59.998 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:54.929 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:00.002 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:33:54.892 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:00.007 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:33:54.938 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:07.790 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:02.721 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:07.795 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:02.727 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:07.801 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:02.691 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:07.802 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:02.734 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:07.804 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:02.694 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:07.805 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:02.736 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:07.816 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:02.748 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:07.821 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:02.711 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:07.826 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:02.758 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:15.010 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:09.941 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:15.013 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:09.944 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:15.024 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:09.955 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:15.061 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:09.951 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:15.065 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:09.955 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:15.075 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:09.964 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:15.090 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:10.022 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:15.100 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:10.031 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:15.109 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:10.041 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:20.027 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:14.958 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:20.031 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:14.962 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:20.046 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:14.977 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:20.079 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:14.969 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:20.084 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:14.974 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:20.086 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:15.017 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:20.089 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:15.021 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:20.094 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:14.984 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:20.101 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:15.032 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:35.093 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:30.024 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:35.094 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:30.025 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:35.111 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:30.042 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:35.148 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:30.037 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:35.148 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:30.038 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:35.149 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:30.081 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:35.157 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:30.089 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:35.163 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:30.053 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:35.172 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:30.104 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:40.076 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:35.007 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:40.078 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:35.009 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:40.088 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:35.019 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:40.125 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:35.014 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:40.128 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:35.017 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:40.131 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:35.062 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:40.132 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:35.063 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:40.140 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:35.030 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:40.143 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:35.074 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:18:48.488 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: Running main() from gmock_main.cc 2024-05-04T18:18:48.488 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-05-04T18:18:48.488 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: [----------] Global test environment set-up. 2024-05-04T18:18:48.488 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-05-04T18:18:48.488 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-05-04T18:18:48.488 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (12430 ms) 2024-05-04T18:18:48.488 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-05-04T18:18:48.488 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (1934047 ms) 2024-05-04T18:18:48.488 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (1946477 ms total) 2024-05-04T18:18:48.488 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: 2024-05-04T18:18:48.488 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: [----------] Global test environment tear-down 2024-05-04T18:18:48.488 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (1946477 ms total) 2024-05-04T18:18:48.489 INFO:tasks.workunit.client.0.smithi019.stdout: ec_io: [ PASSED ] 2 tests. 2024-05-04T18:18:48.492 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38482 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38482 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39145 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39145 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39006 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39006 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38339 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38339 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38405 2024-05-04T18:18:48.493 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38405 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39030 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39030 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38330 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38330 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38350 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38350 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39421 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39421 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38574 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38574 2024-05-04T18:18:48.494 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39488 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39488 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38679 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38679 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39180 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39180 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39539 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39539 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38582 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38582 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39060 2024-05-04T18:18:48.495 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39060 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39375 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39375 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39155 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39155 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39441 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39441 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38971 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38971 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38501 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38501 2024-05-04T18:18:48.496 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39514 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39514 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38836 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38836 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38323 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38323 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38383 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38383 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38739 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38739 2024-05-04T18:18:48.497 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.498 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38423 2024-05-04T18:18:48.498 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38423 2024-05-04T18:18:48.498 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.498 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38884 2024-05-04T18:18:48.498 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38884 2024-05-04T18:18:48.498 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.498 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=38946 2024-05-04T18:18:48.498 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 38946 2024-05-04T18:18:48.498 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.498 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39337 2024-05-04T18:18:48.499 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39337 2024-05-04T18:18:48.499 INFO:tasks.workunit.client.0.smithi019.stderr:+ for t in "${!pids[@]}" 2024-05-04T18:18:48.499 INFO:tasks.workunit.client.0.smithi019.stderr:+ pid=39550 2024-05-04T18:18:48.499 INFO:tasks.workunit.client.0.smithi019.stderr:+ wait 39550 2024-05-04T18:18:48.499 INFO:tasks.workunit.client.0.smithi019.stderr:+ exit 0 2024-05-04T18:18:48.499 INFO:tasks.workunit.client.0.smithi019.stderr:+ cleanup 2024-05-04T18:18:48.499 INFO:tasks.workunit.client.0.smithi019.stderr:+ pkill -P 38317 2024-05-04T18:18:48.502 INFO:tasks.workunit.client.0.smithi019.stderr:+ true 2024-05-04T18:18:48.503 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-04T18:18:48.503 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-04T18:18:48.568 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-05-04T18:18:48.569 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-04T18:18:48.902 DEBUG:teuthology.parallel:result is None 2024-05-04T18:18:48.903 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-04T18:18:48.927 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-04T18:18:48.927 DEBUG:teuthology.orchestra.run.smithi019:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-05-04T18:18:48.996 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-05-04T18:18:48.996 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-04T18:18:49.006 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-04T18:18:49.007 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T18:18:49.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.136+0000 7f4ba96b3640 10 monclient: get_monmap_and_config 2024-05-04T18:18:49.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.136+0000 7f4ba96b3640 10 monclient: build_initial_monmap 2024-05-04T18:18:49.138 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.136+0000 7f4ba96b3640 10 monclient: monmap: 2024-05-04T18:18:49.138 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:49.138 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:49.139 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:49.138747+0000 2024-05-04T18:18:49.139 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:49.138747+0000 2024-05-04T18:18:49.139 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:49.139 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:49.139 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:49.139 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:49.139 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:49.139 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:49.139 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:49.139 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:49.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.137+0000 7f4ba96b3640 1 Processor -- start 2024-05-04T18:18:49.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.137+0000 7f4ba96b3640 1 -- start start 2024-05-04T18:18:49.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.137+0000 7f4ba96b3640 10 monclient: init 2024-05-04T18:18:49.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba96b3640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:49.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba96b3640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-05-04T18:18:49.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba96b3640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 0x7f4ba413c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:49.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba96b3640 10 monclient(hunting): picked mon.noname-b con 0x7f4ba40afe10 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:49.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba96b3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ba413c850 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba96b3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ba4072f90 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba96b3640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ba406fd30 con 0x7f4ba413c850 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba96b3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ba40705a0 con 0x7f4ba4072f90 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba96b3640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba96b3640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba40b01f0 con 0x7f4ba40afe10 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba96b3640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba96b3640 10 monclient(hunting): authenticate will time out at 3351.252841s 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba2ffd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 0x7f4ba413c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba27fc640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f4ba413c850 0x7f4ba4072880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:56428/0 (socket says 172.21.15.19:56428) 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.138+0000 7f4ba27fc640 1 -- 172.21.15.19:0/3145814269 learned_addr learned my addr 172.21.15.19:0/3145814269 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.139+0000 7f4ba2ffd640 10 monclient(hunting): get_auth_request con 0x7f4ba40afe10 auth_method 0 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.139+0000 7f4ba2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.139+0000 7f4ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:49.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.139+0000 7f4ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:49.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.139+0000 7f4ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:49.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.139+0000 7f4ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:49.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.139+0000 7f4ba2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba1ffb640 1 -- 172.21.15.19:0/3145814269 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2233498722 0 0) 0x7f4ba40705a0 con 0x7f4ba4072f90 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba1ffb640 10 monclient(hunting): _negotiate 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba1ffb640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba1ffb640 10 monclient(hunting): my global_id is 22708 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba1ffb640 1 -- 172.21.15.19:0/3145814269 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b90003520 con 0x7f4ba4072f90 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba2ffd640 10 monclient(hunting): handle_auth_done global_id 18758 payload 274 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba2ffd640 1 -- 172.21.15.19:0/3145814269 >> v1:172.21.15.186:6789/0 conn(0x7f4ba4072f90 legacy=0x7f4ba4149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba2ffd640 1 -- 172.21.15.19:0/3145814269 >> v1:172.21.15.19:6789/0 conn(0x7f4ba413c850 legacy=0x7f4ba4072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba2ffd640 10 monclient: _finish_hunting 0 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba2ffd640 1 monclient: found mon.noname-b 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T18:18:49.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.141+0000 7f4ba2ffd640 1 -- 172.21.15.19:0/3145814269 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba4071110 con 0x7f4ba40afe10 2024-05-04T18:18:49.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.143+0000 7f4ba2ffd640 1 --2- 172.21.15.19:0/3145814269 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 0x7f4ba413c310 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00a970 tx=0x7f4b9c02cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=254a3a04c325bc59 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:49.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 1 -- 172.21.15.19:0/3145814269 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c03a070 con 0x7f4ba40afe10 2024-05-04T18:18:49.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 10 monclient: dump: 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 10 monclient: _finish_auth 0 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 10 monclient: _check_auth_tickets 2024-05-04T18:18:49.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 1 -- 172.21.15.19:0/3145814269 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b9c02db20 con 0x7f4ba40afe10 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 1 -- 172.21.15.19:0/3145814269 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c00b040 con 0x7f4ba40afe10 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba96b3640 5 monclient: authenticate success, global_id 18758 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba96b3640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba96b3640 4 set_mon_vals no callback set 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.147+0000 7f4ba1ffb640 10 monclient: dump: 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:49.150 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:49.151 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:49.151 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:49.151 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:49.151 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:49.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.148+0000 7f4ba96b3640 10 monclient: shutdown 2024-05-04T18:18:49.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.148+0000 7f4ba96b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:49.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.148+0000 7f4ba96b3640 1 -- 172.21.15.19:0/3145814269 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 msgr2=0x7f4ba413c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:49.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.148+0000 7f4ba96b3640 1 --2- 172.21.15.19:0/3145814269 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 0x7f4ba413c310 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00a970 tx=0x7f4b9c02cd70 comp rx=0 tx=0).stop 2024-05-04T18:18:49.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.148+0000 7f4ba96b3640 1 -- 172.21.15.19:0/3145814269 shutdown_connections 2024-05-04T18:18:49.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.148+0000 7f4ba96b3640 1 --2- 172.21.15.19:0/3145814269 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 0x7f4ba413c310 unknown :-1 s=CLOSED pgs=3160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.148+0000 7f4ba96b3640 1 -- 172.21.15.19:0/3145814269 >> 172.21.15.19:0/3145814269 conn(0x7f4ba40b4870 msgr2=0x7f4ba40b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.148+0000 7f4ba96b3640 1 -- 172.21.15.19:0/3145814269 shutdown_connections 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.148+0000 7f4ba96b3640 1 -- 172.21.15.19:0/3145814269 wait complete. 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.148+0000 7f4ba96b3640 10 monclient: build_initial_monmap 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.148+0000 7f4ba96b3640 10 monclient: monmap: 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:49.155 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.149+0000 7f4ba96b3640 1 librados: starting msgr at 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.149+0000 7f4ba96b3640 1 librados: starting objecter 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.149+0000 7f4ba96b3640 1 Processor -- start 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.149+0000 7f4ba96b3640 1 -- start start 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.149+0000 7f4ba96b3640 1 librados: setting wanted keys 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.149+0000 7f4ba96b3640 1 librados: calling monclient init 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.149+0000 7f4ba96b3640 10 monclient: init 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ba4072f90 0x7f4ba41cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 10 monclient(hunting): picked mon.noname-b con 0x7f4ba4072f90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 0x7f4ba40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 10 monclient(hunting): picked mon.noname-c con 0x7f4ba40afe10 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ba413c850 0x7f4ba41d7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 10 monclient(hunting): picked mon.noname-a con 0x7f4ba413c850 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:49.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba413cc90 con 0x7f4ba413c850 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4ba4070c30 con 0x7f4ba4072f90 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f4ba40b13a0 con 0x7f4ba40afe10 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba96b3640 10 monclient(hunting): authenticate will time out at 3351.264769s 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba27fc640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 0x7f4ba40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba37fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ba413c850 0x7f4ba41d7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba27fc640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 0x7f4ba40b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:42984/0 (socket says 172.21.15.19:42984) 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba27fc640 1 -- 172.21.15.19:0/1388290118 learned_addr learned my addr 172.21.15.19:0/1388290118 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba27fc640 10 monclient(hunting): get_auth_request con 0x7f4ba40afe10 auth_method 0 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba27fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:49.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.150+0000 7f4ba27fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba2ffd640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ba4072f90 0x7f4ba41cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba37fe640 10 monclient(hunting): get_auth_request con 0x7f4ba413c850 auth_method 0 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba37fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba37fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba2ffd640 10 monclient(hunting): get_auth_request con 0x7f4ba4072f90 auth_method 0 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:49.158 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:49.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:49.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:49.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:49.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.151+0000 7f4ba2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.154+0000 7f4ba27fc640 10 monclient(hunting): handle_auth_done global_id 18761 payload 824 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.154+0000 7f4ba27fc640 1 -- 172.21.15.19:0/1388290118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ba4072f90 msgr2=0x7f4ba41cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.154+0000 7f4ba27fc640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ba4072f90 0x7f4ba41cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.154+0000 7f4ba27fc640 1 -- 172.21.15.19:0/1388290118 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ba413c850 msgr2=0x7f4ba41d7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.154+0000 7f4ba27fc640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ba413c850 0x7f4ba41d7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.154+0000 7f4ba27fc640 10 monclient: _finish_hunting 0 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.154+0000 7f4ba27fc640 1 monclient: found mon.noname-c 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.154+0000 7f4ba27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.154+0000 7f4ba27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.154+0000 7f4ba27fc640 1 -- 172.21.15.19:0/1388290118 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba40b33d0 con 0x7f4ba40afe10 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.156+0000 7f4ba27fc640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 0x7f4ba40b2e90 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c00c3f0 tx=0x7f4b8c00c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.156+0000 7f4b9b7fe640 10 client.?.objecter ms_handle_connect 0x7f4ba40afe10 2024-05-04T18:18:49.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.156+0000 7f4b9b7fe640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:49.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 1 -- 172.21.15.19:0/1388290118 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c017070 con 0x7f4ba40afe10 2024-05-04T18:18:49.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:49.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T18:18:49.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 10 monclient: dump: 2024-05-04T18:18:49.162 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:49.162 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 10 monclient: _finish_auth 0 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 10 monclient: _check_auth_tickets 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4ba96b3640 5 monclient: authenticate success, global_id 18761 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 1 -- 172.21.15.19:0/1388290118 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b8c005ce0 con 0x7f4ba40afe10 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4ba96b3640 10 monclient: _renew_subs 2024-05-04T18:18:49.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4ba96b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4ba96b3640 1 -- 172.21.15.19:0/1388290118 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ba41c83e0 con 0x7f4ba40afe10 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 1 -- 172.21.15.19:0/1388290118 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c0072f0 con 0x7f4ba40afe10 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4b9b7fe640 10 monclient: dump: 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:49.164 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:49.165 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:49.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4ba17fa640 4 set_mon_vals no callback set 2024-05-04T18:18:49.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4ba96b3640 10 client.18761.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:49.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4ba96b3640 10 monclient: _renew_subs 2024-05-04T18:18:49.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4ba96b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:49.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4ba96b3640 1 -- 172.21.15.19:0/1388290118 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ba41d84b0 con 0x7f4ba40afe10 2024-05-04T18:18:49.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.160+0000 7f4ba96b3640 1 librados: init done 2024-05-04T18:18:49.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.161+0000 7f4ba96b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:49.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.161+0000 7f4ba96b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:49.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.161+0000 7f4ba96b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:49.165 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.161+0000 7f4ba96b3640 1 -- 172.21.15.19:0/1388290118 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ba4068b10 con 0x7f4ba40afe10 2024-05-04T18:18:49.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.181+0000 7f4b9b7fe640 1 -- 172.21.15.19:0/1388290118 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4b8c01a070 con 0x7f4ba40afe10 2024-05-04T18:18:49.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.181+0000 7f4b9b7fe640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f4b8406d810 0x7f4b8406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:49.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.181+0000 7f4ba2ffd640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f4b8406d810 0x7f4b8406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:49.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.181+0000 7f4ba2ffd640 10 monclient: get_auth_request con 0x7f4b8406d810 auth_method 0 2024-05-04T18:18:49.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.182+0000 7f4ba2ffd640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f4b8406d810 0x7f4b8406fcd0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c037000 tx=0x7f4b9c00c040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:49.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.182+0000 7f4b9b7fe640 10 client.18761.objecter ms_handle_connect 0x7f4b8406d810 2024-05-04T18:18:49.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.184+0000 7f4b9b7fe640 1 -- 172.21.15.19:0/1388290118 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f4b8c056970 con 0x7f4ba40afe10 2024-05-04T18:18:49.185 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.184+0000 7f4b9b7fe640 10 client.18761.objecter ms_dispatch 0x7f4ba4000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:49.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.184+0000 7f4b9b7fe640 3 client.18761.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:49.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.184+0000 7f4b9b7fe640 3 client.18761.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:49.186 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.184+0000 7f4b9b7fe640 20 client.18761.objecter dump_active .. 0 homeless 2024-05-04T18:18:49.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.456+0000 7f4b9b7fe640 1 -- 172.21.15.19:0/1388290118 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4b8c021020 con 0x7f4ba40afe10 2024-05-04T18:18:49.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.456+0000 7f4b9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:49.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.456+0000 7f4b9b7fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:18:49.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.647+0000 7f4ba96b3640 1 -- 172.21.15.19:0/1388290118 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4ba40678c0 con 0x7f4b8406d810 2024-05-04T18:18:49.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.651+0000 7f4b9b7fe640 1 -- 172.21.15.19:0/1388290118 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116259 (secure 0 0 0) 0x7f4b9c061640 con 0x7f4b8406d810 2024-05-04T18:18:49.653 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T18:18:49.654 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-05-04T18:18:49.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 10 librados: watch_flush enter 2024-05-04T18:18:49.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 10 librados: watch_flush exit 2024-05-04T18:18:49.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 20 client.18761.objecter shutdown clearing up homeless session... 2024-05-04T18:18:49.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 10 client.18761.objecter successfully canceled tick 2024-05-04T18:18:49.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 -- 172.21.15.19:0/1388290118 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f4b8406d810 msgr2=0x7f4b8406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:49.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f4b8406d810 0x7f4b8406fcd0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c037000 tx=0x7f4b9c00c040 comp rx=0 tx=0).stop 2024-05-04T18:18:49.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 10 monclient: shutdown 2024-05-04T18:18:49.656 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 -- 172.21.15.19:0/1388290118 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 msgr2=0x7f4ba40b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 0x7f4ba40b2e90 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c00c3f0 tx=0x7f4b8c00c8b0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 -- 172.21.15.19:0/1388290118 shutdown_connections 2024-05-04T18:18:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f4b8406d810 0x7f4b8406fcd0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4ba413c850 0x7f4ba41d7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4ba40afe10 0x7f4ba40b2e90 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 --2- 172.21.15.19:0/1388290118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ba4072f90 0x7f4ba41cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 -- 172.21.15.19:0/1388290118 >> 172.21.15.19:0/1388290118 conn(0x7f4ba40b4870 msgr2=0x7f4ba41cdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 -- 172.21.15.19:0/1388290118 shutdown_connections 2024-05-04T18:18:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 -- 172.21.15.19:0/1388290118 wait complete. 2024-05-04T18:18:49.657 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.653+0000 7f4ba96b3640 1 librados: shutdown 2024-05-04T18:18:49.665 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":2204,"stamp":"2024-05-04T18:18:49.526620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":542360,"kb_used_data":22300,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754432360,"statfs":{"total":773094113280,"available":772538736640,"internally_reserved":0,"allocated":22835200,"data_stored":8395419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":529714736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":175,"apply_latency_ms":175,"commit_latency_ns":175000000,"apply_latency_ns":175000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001257"},"pg_stats":[{"pgid":"14.1e","version":"0'0","reported_seq":2262,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.646924+0000","last_change":"2024-05-04T17:46:46.574386+0000","last_active":"2024-05-04T18:18:47.646924+0000","last_peered":"2024-05-04T18:18:47.646924+0000","last_clean":"2024-05-04T18:18:47.646924+0000","last_became_active":"2024-05-04T17:46:46.518637+0000","last_became_peered":"2024-05-04T17:46:46.518637+0000","last_unstale":"2024-05-04T18:18:47.646924+0000","last_undegraded":"2024-05-04T18:18:47.646924+0000","last_fullsized":"2024-05-04T18:18:47.646924+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:40:26.227815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":2263,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.453243+0000","last_change":"2024-05-04T18:00:26.429131+0000","last_active":"2024-05-04T18:18:47.453243+0000","last_peered":"2024-05-04T18:18:47.453243+0000","last_clean":"2024-05-04T18:18:47.453243+0000","last_became_active":"2024-05-04T18:00:26.421590+0000","last_became_peered":"2024-05-04T18:00:26.421590+0000","last_unstale":"2024-05-04T18:18:47.453243+0000","last_undegraded":"2024-05-04T18:18:47.453243+0000","last_fullsized":"2024-05-04T18:18:47.453243+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:10:52.211055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":2267,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.591503+0000","last_change":"2024-05-04T17:47:40.091354+0000","last_active":"2024-05-04T18:18:47.591503+0000","last_peered":"2024-05-04T18:18:47.591503+0000","last_clean":"2024-05-04T18:18:47.591503+0000","last_became_active":"2024-05-04T17:47:40.078079+0000","last_became_peered":"2024-05-04T17:47:40.078079+0000","last_unstale":"2024-05-04T18:18:47.591503+0000","last_undegraded":"2024-05-04T18:18:47.591503+0000","last_fullsized":"2024-05-04T18:18:47.591503+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:08:09.078519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":2278,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.250747+0000","last_change":"2024-05-04T17:46:53.620558+0000","last_active":"2024-05-04T18:17:25.250747+0000","last_peered":"2024-05-04T18:17:25.250747+0000","last_clean":"2024-05-04T18:17:25.250747+0000","last_became_active":"2024-05-04T17:46:53.602969+0000","last_became_peered":"2024-05-04T17:46:53.602969+0000","last_unstale":"2024-05-04T18:17:25.250747+0000","last_undegraded":"2024-05-04T18:17:25.250747+0000","last_fullsized":"2024-05-04T18:17:25.250747+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:00:17.742135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":2265,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.553741+0000","last_change":"2024-05-04T18:00:25.071880+0000","last_active":"2024-05-04T18:18:47.553741+0000","last_peered":"2024-05-04T18:18:47.553741+0000","last_clean":"2024-05-04T18:18:47.553741+0000","last_became_active":"2024-05-04T18:00:25.064688+0000","last_became_peered":"2024-05-04T18:00:25.064688+0000","last_unstale":"2024-05-04T18:18:47.553741+0000","last_undegraded":"2024-05-04T18:18:47.553741+0000","last_fullsized":"2024-05-04T18:18:47.553741+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:58:15.875261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":2281,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.261803+0000","last_change":"2024-05-04T17:46:53.705592+0000","last_active":"2024-05-04T18:17:25.261803+0000","last_peered":"2024-05-04T18:17:25.261803+0000","last_clean":"2024-05-04T18:17:25.261803+0000","last_became_active":"2024-05-04T17:46:53.682041+0000","last_became_peered":"2024-05-04T17:46:53.682041+0000","last_unstale":"2024-05-04T18:17:25.261803+0000","last_undegraded":"2024-05-04T18:17:25.261803+0000","last_fullsized":"2024-05-04T18:17:25.261803+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:09:58.581895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":2287,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.236439+0000","last_change":"2024-05-04T17:46:35.450534+0000","last_active":"2024-05-04T18:17:25.236439+0000","last_peered":"2024-05-04T18:17:25.236439+0000","last_clean":"2024-05-04T18:17:25.236439+0000","last_became_active":"2024-05-04T17:46:35.438066+0000","last_became_peered":"2024-05-04T17:46:35.438066+0000","last_unstale":"2024-05-04T18:17:25.236439+0000","last_undegraded":"2024-05-04T18:17:25.236439+0000","last_fullsized":"2024-05-04T18:17:25.236439+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:28:24.859168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1","version":"105'7","reported_seq":2269,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.435333+0000","last_change":"2024-05-04T17:46:53.697892+0000","last_active":"2024-05-04T18:18:47.435333+0000","last_peered":"2024-05-04T18:18:47.435333+0000","last_clean":"2024-05-04T18:18:47.435333+0000","last_became_active":"2024-05-04T17:46:53.692194+0000","last_became_peered":"2024-05-04T17:46:53.692194+0000","last_unstale":"2024-05-04T18:18:47.435333+0000","last_undegraded":"2024-05-04T18:18:47.435333+0000","last_fullsized":"2024-05-04T18:18:47.435333+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:29:13.161473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":2279,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.321514+0000","last_change":"2024-05-04T17:46:32.428706+0000","last_active":"2024-05-04T18:18:46.321514+0000","last_peered":"2024-05-04T18:18:46.321514+0000","last_clean":"2024-05-04T18:18:46.321514+0000","last_became_active":"2024-05-04T17:46:32.422070+0000","last_became_peered":"2024-05-04T17:46:32.422070+0000","last_unstale":"2024-05-04T18:18:46.321514+0000","last_undegraded":"2024-05-04T18:18:46.321514+0000","last_fullsized":"2024-05-04T18:18:46.321514+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:08:42.163294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":2240,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.624948+0000","last_change":"2024-05-04T17:47:33.488923+0000","last_active":"2024-05-04T18:18:47.624948+0000","last_peered":"2024-05-04T18:18:47.624948+0000","last_clean":"2024-05-04T18:18:47.624948+0000","last_became_active":"2024-05-04T17:47:33.470216+0000","last_became_peered":"2024-05-04T17:47:33.470216+0000","last_unstale":"2024-05-04T18:18:47.624948+0000","last_undegraded":"2024-05-04T18:18:47.624948+0000","last_fullsized":"2024-05-04T18:18:47.624948+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:31:16.602746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":2247,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.572078+0000","last_change":"2024-05-04T17:46:55.297891+0000","last_active":"2024-05-04T18:18:47.572078+0000","last_peered":"2024-05-04T18:18:47.572078+0000","last_clean":"2024-05-04T18:18:47.572078+0000","last_became_active":"2024-05-04T17:46:55.287626+0000","last_became_peered":"2024-05-04T17:46:55.287626+0000","last_unstale":"2024-05-04T18:18:47.572078+0000","last_undegraded":"2024-05-04T18:18:47.572078+0000","last_fullsized":"2024-05-04T18:18:47.572078+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:47:08.351184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":2280,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.317119+0000","last_change":"2024-05-04T17:46:35.598755+0000","last_active":"2024-05-04T18:18:46.317119+0000","last_peered":"2024-05-04T18:18:46.317119+0000","last_clean":"2024-05-04T18:18:46.317119+0000","last_became_active":"2024-05-04T17:46:35.592584+0000","last_became_peered":"2024-05-04T17:46:35.592584+0000","last_unstale":"2024-05-04T18:18:46.317119+0000","last_undegraded":"2024-05-04T18:18:46.317119+0000","last_fullsized":"2024-05-04T18:18:46.317119+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:35:42.158774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":2240,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.627053+0000","last_change":"2024-05-04T17:47:04.885267+0000","last_active":"2024-05-04T18:18:47.627053+0000","last_peered":"2024-05-04T18:18:47.627053+0000","last_clean":"2024-05-04T18:18:47.627053+0000","last_became_active":"2024-05-04T17:47:04.857381+0000","last_became_peered":"2024-05-04T17:47:04.857381+0000","last_unstale":"2024-05-04T18:18:47.627053+0000","last_undegraded":"2024-05-04T18:18:47.627053+0000","last_fullsized":"2024-05-04T18:18:47.627053+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:09:14.931257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"18'76","reported_seq":2279,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.570917+0000","last_change":"2024-05-04T17:45:18.398964+0000","last_active":"2024-05-04T18:18:47.570917+0000","last_peered":"2024-05-04T18:18:47.570917+0000","last_clean":"2024-05-04T18:18:47.570917+0000","last_became_active":"2024-05-04T17:45:17.720848+0000","last_became_peered":"2024-05-04T17:45:17.720848+0000","last_unstale":"2024-05-04T18:18:47.570917+0000","last_undegraded":"2024-05-04T18:18:47.570917+0000","last_fullsized":"2024-05-04T18:18:47.570917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:03.919442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:03.919442+0000","last_clean_scrub_stamp":"2024-05-04T17:45:03.919442+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:32:22.959179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2279,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.578512+0000","last_change":"2024-05-04T17:45:23.593473+0000","last_active":"2024-05-04T18:18:47.578512+0000","last_peered":"2024-05-04T18:18:47.578512+0000","last_clean":"2024-05-04T18:18:47.578512+0000","last_became_active":"2024-05-04T17:45:23.583946+0000","last_became_peered":"2024-05-04T17:45:23.583946+0000","last_unstale":"2024-05-04T18:18:47.578512+0000","last_undegraded":"2024-05-04T18:18:47.578512+0000","last_fullsized":"2024-05-04T18:18:47.578512+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:19:20.455748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":2280,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.379217+0000","last_change":"2024-05-04T17:46:45.398622+0000","last_active":"2024-05-04T18:18:46.379217+0000","last_peered":"2024-05-04T18:18:46.379217+0000","last_clean":"2024-05-04T18:18:46.379217+0000","last_became_active":"2024-05-04T17:46:45.387152+0000","last_became_peered":"2024-05-04T17:46:45.387152+0000","last_unstale":"2024-05-04T18:18:46.379217+0000","last_undegraded":"2024-05-04T18:18:46.379217+0000","last_fullsized":"2024-05-04T18:18:46.379217+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:51:33.866454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2282,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.569454+0000","last_change":"2024-05-04T17:45:23.292542+0000","last_active":"2024-05-04T18:18:47.569454+0000","last_peered":"2024-05-04T18:18:47.569454+0000","last_clean":"2024-05-04T18:18:47.569454+0000","last_became_active":"2024-05-04T17:45:23.262086+0000","last_became_peered":"2024-05-04T17:45:23.262086+0000","last_unstale":"2024-05-04T18:18:47.569454+0000","last_undegraded":"2024-05-04T18:18:47.569454+0000","last_fullsized":"2024-05-04T18:18:47.569454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:16:49.542097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":2265,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.633827+0000","last_change":"2024-05-04T17:47:14.147039+0000","last_active":"2024-05-04T18:18:47.633827+0000","last_peered":"2024-05-04T18:18:47.633827+0000","last_clean":"2024-05-04T18:18:47.633827+0000","last_became_active":"2024-05-04T17:47:14.130381+0000","last_became_peered":"2024-05-04T17:47:14.130381+0000","last_unstale":"2024-05-04T18:18:47.633827+0000","last_undegraded":"2024-05-04T18:18:47.633827+0000","last_fullsized":"2024-05-04T18:18:47.633827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:56:09.055998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2272,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.438925+0000","last_change":"2024-05-04T17:45:23.301096+0000","last_active":"2024-05-04T18:18:47.438925+0000","last_peered":"2024-05-04T18:18:47.438925+0000","last_clean":"2024-05-04T18:18:47.438925+0000","last_became_active":"2024-05-04T17:45:23.265024+0000","last_became_peered":"2024-05-04T17:45:23.265024+0000","last_unstale":"2024-05-04T18:18:47.438925+0000","last_undegraded":"2024-05-04T18:18:47.438925+0000","last_fullsized":"2024-05-04T18:18:47.438925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:48:10.154363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":2243,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.640584+0000","last_change":"2024-05-04T18:00:26.429267+0000","last_active":"2024-05-04T18:18:47.640584+0000","last_peered":"2024-05-04T18:18:47.640584+0000","last_clean":"2024-05-04T18:18:47.640584+0000","last_became_active":"2024-05-04T18:00:26.421083+0000","last_became_peered":"2024-05-04T18:00:26.421083+0000","last_unstale":"2024-05-04T18:18:47.640584+0000","last_undegraded":"2024-05-04T18:18:47.640584+0000","last_fullsized":"2024-05-04T18:18:47.640584+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:26:20.262350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2280,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.595840+0000","last_change":"2024-05-04T17:45:23.491691+0000","last_active":"2024-05-04T18:18:47.595840+0000","last_peered":"2024-05-04T18:18:47.595840+0000","last_clean":"2024-05-04T18:18:47.595840+0000","last_became_active":"2024-05-04T17:45:23.480170+0000","last_became_peered":"2024-05-04T17:45:23.480170+0000","last_unstale":"2024-05-04T18:18:47.595840+0000","last_undegraded":"2024-05-04T18:18:47.595840+0000","last_fullsized":"2024-05-04T18:18:47.595840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:18:07.981076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":2285,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.237462+0000","last_change":"2024-05-04T17:46:35.456577+0000","last_active":"2024-05-04T18:17:25.237462+0000","last_peered":"2024-05-04T18:17:25.237462+0000","last_clean":"2024-05-04T18:17:25.237462+0000","last_became_active":"2024-05-04T17:46:35.445021+0000","last_became_peered":"2024-05-04T17:46:35.445021+0000","last_unstale":"2024-05-04T18:17:25.237462+0000","last_undegraded":"2024-05-04T18:17:25.237462+0000","last_fullsized":"2024-05-04T18:17:25.237462+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:46:31.130508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2290,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.381308+0000","last_change":"2024-05-04T17:45:23.488141+0000","last_active":"2024-05-04T18:18:46.381308+0000","last_peered":"2024-05-04T18:18:46.381308+0000","last_clean":"2024-05-04T18:18:46.381308+0000","last_became_active":"2024-05-04T17:45:23.481564+0000","last_became_peered":"2024-05-04T17:45:23.481564+0000","last_unstale":"2024-05-04T18:18:46.381308+0000","last_undegraded":"2024-05-04T18:18:46.381308+0000","last_fullsized":"2024-05-04T18:18:46.381308+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:23:41.989802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":2243,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.234068+0000","last_change":"2024-05-04T17:46:37.667579+0000","last_active":"2024-05-04T18:17:25.234068+0000","last_peered":"2024-05-04T18:17:25.234068+0000","last_clean":"2024-05-04T18:17:25.234068+0000","last_became_active":"2024-05-04T17:46:37.661672+0000","last_became_peered":"2024-05-04T17:46:37.661672+0000","last_unstale":"2024-05-04T18:17:25.234068+0000","last_undegraded":"2024-05-04T18:17:25.234068+0000","last_fullsized":"2024-05-04T18:17:25.234068+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:58:33.429360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2281,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.629869+0000","last_change":"2024-05-04T17:45:23.379564+0000","last_active":"2024-05-04T18:18:47.629869+0000","last_peered":"2024-05-04T18:18:47.629869+0000","last_clean":"2024-05-04T18:18:47.629869+0000","last_became_active":"2024-05-04T17:45:23.370117+0000","last_became_peered":"2024-05-04T17:45:23.370117+0000","last_unstale":"2024-05-04T18:18:47.629869+0000","last_undegraded":"2024-05-04T18:18:47.629869+0000","last_fullsized":"2024-05-04T18:18:47.629869+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:39:45.488747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":2259,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.560915+0000","last_change":"2024-05-04T17:47:00.057580+0000","last_active":"2024-05-04T18:18:47.560915+0000","last_peered":"2024-05-04T18:18:47.560915+0000","last_clean":"2024-05-04T18:18:47.560915+0000","last_became_active":"2024-05-04T17:47:00.051980+0000","last_became_peered":"2024-05-04T17:47:00.051980+0000","last_unstale":"2024-05-04T18:18:47.560915+0000","last_undegraded":"2024-05-04T18:18:47.560915+0000","last_fullsized":"2024-05-04T18:18:47.560915+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:01:05.388827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2295,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.258047+0000","last_change":"2024-05-04T17:45:23.713140+0000","last_active":"2024-05-04T18:17:25.258047+0000","last_peered":"2024-05-04T18:17:25.258047+0000","last_clean":"2024-05-04T18:17:25.258047+0000","last_became_active":"2024-05-04T17:45:23.686812+0000","last_became_peered":"2024-05-04T17:45:23.686812+0000","last_unstale":"2024-05-04T18:17:25.258047+0000","last_undegraded":"2024-05-04T18:17:25.258047+0000","last_fullsized":"2024-05-04T18:17:25.258047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:56:49.147299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":2269,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.583227+0000","last_change":"2024-05-04T17:47:40.993471+0000","last_active":"2024-05-04T18:18:47.583227+0000","last_peered":"2024-05-04T18:18:47.583227+0000","last_clean":"2024-05-04T18:18:47.583227+0000","last_became_active":"2024-05-04T17:47:40.988027+0000","last_became_peered":"2024-05-04T17:47:40.988027+0000","last_unstale":"2024-05-04T18:18:47.583227+0000","last_undegraded":"2024-05-04T18:18:47.583227+0000","last_fullsized":"2024-05-04T18:18:47.583227+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:41:52.420496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2276,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.650826+0000","last_change":"2024-05-04T17:45:23.672655+0000","last_active":"2024-05-04T18:18:47.650826+0000","last_peered":"2024-05-04T18:18:47.650826+0000","last_clean":"2024-05-04T18:18:47.650826+0000","last_became_active":"2024-05-04T17:45:23.667080+0000","last_became_peered":"2024-05-04T17:45:23.667080+0000","last_unstale":"2024-05-04T18:18:47.650826+0000","last_undegraded":"2024-05-04T18:18:47.650826+0000","last_fullsized":"2024-05-04T18:18:47.650826+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:43:12.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":2264,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.573341+0000","last_change":"2024-05-04T17:46:40.940632+0000","last_active":"2024-05-04T18:18:47.573341+0000","last_peered":"2024-05-04T18:18:47.573341+0000","last_clean":"2024-05-04T18:18:47.573341+0000","last_became_active":"2024-05-04T17:46:40.915492+0000","last_became_peered":"2024-05-04T17:46:40.915492+0000","last_unstale":"2024-05-04T18:18:47.573341+0000","last_undegraded":"2024-05-04T18:18:47.573341+0000","last_fullsized":"2024-05-04T18:18:47.573341+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:43:04.514591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":2272,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.312727+0000","last_change":"2024-05-04T17:46:53.791115+0000","last_active":"2024-05-04T18:18:46.312727+0000","last_peered":"2024-05-04T18:18:46.312727+0000","last_clean":"2024-05-04T18:18:46.312727+0000","last_became_active":"2024-05-04T17:46:53.785221+0000","last_became_peered":"2024-05-04T17:46:53.785221+0000","last_unstale":"2024-05-04T18:18:46.312727+0000","last_undegraded":"2024-05-04T18:18:46.312727+0000","last_fullsized":"2024-05-04T18:18:46.312727+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:28:11.181195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":2274,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.580658+0000","last_change":"2024-05-04T17:46:43.226742+0000","last_active":"2024-05-04T18:18:47.580658+0000","last_peered":"2024-05-04T18:18:47.580658+0000","last_clean":"2024-05-04T18:18:47.580658+0000","last_became_active":"2024-05-04T17:46:43.193128+0000","last_became_peered":"2024-05-04T17:46:43.193128+0000","last_unstale":"2024-05-04T18:18:47.580658+0000","last_undegraded":"2024-05-04T18:18:47.580658+0000","last_fullsized":"2024-05-04T18:18:47.580658+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:52:43.388505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":2283,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.239474+0000","last_change":"2024-05-04T17:46:42.337619+0000","last_active":"2024-05-04T18:17:25.239474+0000","last_peered":"2024-05-04T18:17:25.239474+0000","last_clean":"2024-05-04T18:17:25.239474+0000","last_became_active":"2024-05-04T17:46:42.332321+0000","last_became_peered":"2024-05-04T17:46:42.332321+0000","last_unstale":"2024-05-04T18:17:25.239474+0000","last_undegraded":"2024-05-04T18:17:25.239474+0000","last_fullsized":"2024-05-04T18:17:25.239474+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:45:24.298610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":2262,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.637251+0000","last_change":"2024-05-04T17:47:27.489125+0000","last_active":"2024-05-04T18:18:47.637251+0000","last_peered":"2024-05-04T18:18:47.637251+0000","last_clean":"2024-05-04T18:18:47.637251+0000","last_became_active":"2024-05-04T17:47:27.462872+0000","last_became_peered":"2024-05-04T17:47:27.462872+0000","last_unstale":"2024-05-04T18:18:47.637251+0000","last_undegraded":"2024-05-04T18:18:47.637251+0000","last_fullsized":"2024-05-04T18:18:47.637251+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:31:19.507264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":2241,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.620547+0000","last_change":"2024-05-04T17:47:04.137847+0000","last_active":"2024-05-04T18:18:47.620547+0000","last_peered":"2024-05-04T18:18:47.620547+0000","last_clean":"2024-05-04T18:18:47.620547+0000","last_became_active":"2024-05-04T17:47:04.099185+0000","last_became_peered":"2024-05-04T17:47:04.099185+0000","last_unstale":"2024-05-04T18:18:47.620547+0000","last_undegraded":"2024-05-04T18:18:47.620547+0000","last_fullsized":"2024-05-04T18:18:47.620547+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:52:57.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":2272,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.270653+0000","last_change":"2024-05-04T17:47:15.314211+0000","last_active":"2024-05-04T18:17:25.270653+0000","last_peered":"2024-05-04T18:17:25.270653+0000","last_clean":"2024-05-04T18:17:25.270653+0000","last_became_active":"2024-05-04T17:47:15.307302+0000","last_became_peered":"2024-05-04T17:47:15.307302+0000","last_unstale":"2024-05-04T18:17:25.270653+0000","last_undegraded":"2024-05-04T18:17:25.270653+0000","last_fullsized":"2024-05-04T18:17:25.270653+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:21:22.778712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":2267,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.627221+0000","last_change":"2024-05-04T17:47:40.632890+0000","last_active":"2024-05-04T18:18:47.627221+0000","last_peered":"2024-05-04T18:18:47.627221+0000","last_clean":"2024-05-04T18:18:47.627221+0000","last_became_active":"2024-05-04T17:47:40.627379+0000","last_became_peered":"2024-05-04T17:47:40.627379+0000","last_unstale":"2024-05-04T18:18:47.627221+0000","last_undegraded":"2024-05-04T18:18:47.627221+0000","last_fullsized":"2024-05-04T18:18:47.627221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:24:44.506896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":2241,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.590521+0000","last_change":"2024-05-04T17:47:33.306252+0000","last_active":"2024-05-04T18:18:47.590521+0000","last_peered":"2024-05-04T18:18:47.590521+0000","last_clean":"2024-05-04T18:18:47.590521+0000","last_became_active":"2024-05-04T17:47:33.296115+0000","last_became_peered":"2024-05-04T17:47:33.296115+0000","last_unstale":"2024-05-04T18:18:47.590521+0000","last_undegraded":"2024-05-04T18:18:47.590521+0000","last_fullsized":"2024-05-04T18:18:47.590521+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:49:52.193030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":2275,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.546231+0000","last_change":"2024-05-04T17:46:42.745978+0000","last_active":"2024-05-04T18:18:47.546231+0000","last_peered":"2024-05-04T18:18:47.546231+0000","last_clean":"2024-05-04T18:18:47.546231+0000","last_became_active":"2024-05-04T17:46:42.724604+0000","last_became_peered":"2024-05-04T17:46:42.724604+0000","last_unstale":"2024-05-04T18:18:47.546231+0000","last_undegraded":"2024-05-04T18:18:47.546231+0000","last_fullsized":"2024-05-04T18:18:47.546231+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:52:59.865599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":2279,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.288621+0000","last_change":"2024-05-04T17:46:45.356387+0000","last_active":"2024-05-04T18:18:46.288621+0000","last_peered":"2024-05-04T18:18:46.288621+0000","last_clean":"2024-05-04T18:18:46.288621+0000","last_became_active":"2024-05-04T17:46:45.347368+0000","last_became_peered":"2024-05-04T17:46:45.347368+0000","last_unstale":"2024-05-04T18:18:46.288621+0000","last_undegraded":"2024-05-04T18:18:46.288621+0000","last_fullsized":"2024-05-04T18:18:46.288621+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:10:48.903095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":2268,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.570558+0000","last_change":"2024-05-04T17:47:38.797623+0000","last_active":"2024-05-04T18:18:47.570558+0000","last_peered":"2024-05-04T18:18:47.570558+0000","last_clean":"2024-05-04T18:18:47.570558+0000","last_became_active":"2024-05-04T17:47:38.791956+0000","last_became_peered":"2024-05-04T17:47:38.791956+0000","last_unstale":"2024-05-04T18:18:47.570558+0000","last_undegraded":"2024-05-04T18:18:47.570558+0000","last_fullsized":"2024-05-04T18:18:47.570558+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:47:30.587754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313621,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66692,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305148,"statfs":{"total":96636764160,"available":96568471552,"internally_reserved":0,"allocated":3166208,"data_stored":1353769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[]},{"osd":0,"up_from":20,"seq":85899346321,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66508,"kb_used_data":3152,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305332,"statfs":{"total":96636764160,"available":96568659968,"internally_reserved":0,"allocated":3227648,"data_stored":1224566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66272,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305568,"statfs":{"total":96636764160,"available":96568901632,"internally_reserved":0,"allocated":2985984,"data_stored":1082307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604379030,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66268,"kb_used_data":2912,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305572,"statfs":{"total":96636764160,"available":96568905728,"internally_reserved":0,"allocated":2981888,"data_stored":1082302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":16,"seq":68719477133,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69980,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301860,"statfs":{"total":96636764160,"available":96565104640,"internally_reserved":0,"allocated":2523136,"data_stored":818417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542547,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66064,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305776,"statfs":{"total":96636764160,"available":96569114624,"internally_reserved":0,"allocated":2523136,"data_stored":814326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542543,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69976,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301864,"statfs":{"total":96636764160,"available":96565108736,"internally_reserved":0,"allocated":2519040,"data_stored":814321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575251,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70600,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301240,"statfs":{"total":96636764160,"available":96564469760,"internally_reserved":0,"allocated":2908160,"data_stored":1205411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T18:18:49.668 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T18:18:49.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.766+0000 7f4be9a8c640 10 monclient: get_monmap_and_config 2024-05-04T18:18:49.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.766+0000 7f4be9a8c640 10 monclient: build_initial_monmap 2024-05-04T18:18:49.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.766+0000 7f4be9a8c640 10 monclient: monmap: 2024-05-04T18:18:49.768 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:49.768 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:49.768 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:49.768602+0000 2024-05-04T18:18:49.768 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:49.768602+0000 2024-05-04T18:18:49.768 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:49.768 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:49.769 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:49.769 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:49.769 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:49.769 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:49.769 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:49.769 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:49.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.767+0000 7f4be9a8c640 1 Processor -- start 2024-05-04T18:18:49.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.767+0000 7f4be9a8c640 1 -- start start 2024-05-04T18:18:49.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.767+0000 7f4be9a8c640 10 monclient: init 2024-05-04T18:18:49.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:49.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-05-04T18:18:49.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4be40b2690 0x7f4be40b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:49.771 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 10 monclient(hunting): picked mon.noname-b con 0x7f4be40b2690 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4be413fd70 0x7f4be41401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 10 monclient(hunting): picked mon.noname-c con 0x7f4be413fd70 addr v2:172.21.15.186:3300/0 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4be4142c70 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be41406f0 con 0x7f4be4142c70 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f4be40b2a70 con 0x7f4be413fd70 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f4be40b1a20 con 0x7f4be40b2690 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be9a8c640 10 monclient(hunting): authenticate will time out at 3351.882837s 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be37fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4be40b2690 0x7f4be40b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be2ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4be413fd70 0x7f4be41401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be3fff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f4be4142c70 0x7f4be41430d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:56438/0 (socket says 172.21.15.19:56438) 2024-05-04T18:18:49.772 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.768+0000 7f4be3fff640 1 -- 172.21.15.19:0/594267926 learned_addr learned my addr 172.21.15.19:0/594267926 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be37fe640 10 monclient(hunting): get_auth_request con 0x7f4be40b2690 auth_method 0 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be37fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be2ffd640 10 monclient(hunting): get_auth_request con 0x7f4be413fd70 auth_method 0 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be2ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:49.773 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.769+0000 7f4be37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:49.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.771+0000 7f4be2ffd640 10 monclient(hunting): handle_auth_done global_id 22714 payload 274 2024-05-04T18:18:49.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.771+0000 7f4be2ffd640 1 -- 172.21.15.19:0/594267926 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4be40b2690 msgr2=0x7f4be40b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:49.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.771+0000 7f4be2ffd640 1 --2- 172.21.15.19:0/594267926 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4be40b2690 0x7f4be40b10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.771+0000 7f4be2ffd640 1 -- 172.21.15.19:0/594267926 >> v1:172.21.15.19:6789/0 conn(0x7f4be4142c70 legacy=0x7f4be41430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:49.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.771+0000 7f4be2ffd640 10 monclient: _finish_hunting 0 2024-05-04T18:18:49.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.771+0000 7f4be2ffd640 1 monclient: found mon.noname-c 2024-05-04T18:18:49.774 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.771+0000 7f4be2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:49.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.771+0000 7f4be2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T18:18:49.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.771+0000 7f4be2ffd640 1 -- 172.21.15.19:0/594267926 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be40b1620 con 0x7f4be413fd70 2024-05-04T18:18:49.775 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.773+0000 7f4be2ffd640 1 --2- 172.21.15.19:0/594267926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4be413fd70 0x7f4be41401b0 secure :-1 s=READY pgs=5817 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0097a0 tx=0x7f4bcc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bf65341a1d1429a server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:49.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.777+0000 7f4be27fc640 1 -- 172.21.15.19:0/594267926 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc0315a0 con 0x7f4be413fd70 2024-05-04T18:18:49.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.777+0000 7f4be27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:49.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.777+0000 7f4be27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T18:18:49.779 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.777+0000 7f4be27fc640 10 monclient: dump: 2024-05-04T18:18:49.779 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:49.779 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:49.779 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.779 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.777+0000 7f4be27fc640 10 monclient: _finish_auth 0 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.777+0000 7f4be27fc640 10 monclient: _check_auth_tickets 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.777+0000 7f4be27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 5 monclient: authenticate success, global_id 22714 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.777+0000 7f4be27fc640 1 -- 172.21.15.19:0/594267926 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bcc037070 con 0x7f4be413fd70 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be27fc640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:49.780 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 4 set_mon_vals no callback set 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be27fc640 1 -- 172.21.15.19:0/594267926 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc031960 con 0x7f4be413fd70 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be27fc640 10 monclient: dump: 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:49.781 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 10 monclient: shutdown 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 1 -- 172.21.15.19:0/594267926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4be413fd70 msgr2=0x7f4be41401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 1 --2- 172.21.15.19:0/594267926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4be413fd70 0x7f4be41401b0 secure :-1 s=READY pgs=5817 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0097a0 tx=0x7f4bcc02b840 comp rx=0 tx=0).stop 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 1 -- 172.21.15.19:0/594267926 shutdown_connections 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 1 --2- 172.21.15.19:0/594267926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4be413fd70 0x7f4be41401b0 unknown :-1 s=CLOSED pgs=5817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 1 --2- 172.21.15.19:0/594267926 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4be40b2690 0x7f4be40b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 1 -- 172.21.15.19:0/594267926 >> 172.21.15.19:0/594267926 conn(0x7f4be40b5840 msgr2=0x7f4be40b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 1 -- 172.21.15.19:0/594267926 shutdown_connections 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.778+0000 7f4be9a8c640 1 -- 172.21.15.19:0/594267926 wait complete. 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.779+0000 7f4be9a8c640 10 monclient: build_initial_monmap 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.779+0000 7f4be9a8c640 10 monclient: monmap: 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:49.785 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.779+0000 7f4be9a8c640 1 librados: starting msgr at 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.779+0000 7f4be9a8c640 1 librados: starting objecter 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.779+0000 7f4be9a8c640 1 Processor -- start 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 1 -- start start 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 1 librados: setting wanted keys 2024-05-04T18:18:49.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 1 librados: calling monclient init 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 10 monclient: init 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4be40b2690 0x7f4be4142060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 10 monclient(hunting): picked mon.noname-b con 0x7f4be40b2690 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4be413fd70 0x7f4be4137fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 10 monclient(hunting): picked mon.noname-c con 0x7f4be413fd70 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4be4142c70 0x7f4be4138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 10 monclient(hunting): picked mon.noname-a con 0x7f4be4142c70 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f4be4073800 con 0x7f4be4142c70 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4be4073680 con 0x7f4be40b2690 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f4be414d390 con 0x7f4be413fd70 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:49.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.780+0000 7f4be9a8c640 10 monclient(hunting): authenticate will time out at 3351.895278s 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be2ffd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4be413fd70 0x7f4be4137fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4be40b2690 0x7f4be4142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be3fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4be4142c70 0x7f4be4138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be2ffd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4be413fd70 0x7f4be4137fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:43002/0 (socket says 172.21.15.19:43002) 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be2ffd640 1 -- 172.21.15.19:0/1189369266 learned_addr learned my addr 172.21.15.19:0/1189369266 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be2ffd640 10 monclient(hunting): get_auth_request con 0x7f4be413fd70 auth_method 0 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be2ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be3fff640 10 monclient(hunting): get_auth_request con 0x7f4be4142c70 auth_method 0 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be3fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be3fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be37fe640 10 monclient(hunting): get_auth_request con 0x7f4be40b2690 auth_method 0 2024-05-04T18:18:49.788 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be37fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.781+0000 7f4be37fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.782+0000 7f4be2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.782+0000 7f4be2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.782+0000 7f4be2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.782+0000 7f4be3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.782+0000 7f4be3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.782+0000 7f4be3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.782+0000 7f4be37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.782+0000 7f4be37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:49.789 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.782+0000 7f4be37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:49.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.784+0000 7f4be2ffd640 10 monclient(hunting): handle_auth_done global_id 18767 payload 824 2024-05-04T18:18:49.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.784+0000 7f4be2ffd640 1 -- 172.21.15.19:0/1189369266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4be40b2690 msgr2=0x7f4be4142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:49.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.784+0000 7f4be2ffd640 1 --2- 172.21.15.19:0/1189369266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4be40b2690 0x7f4be4142060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.784+0000 7f4be2ffd640 1 -- 172.21.15.19:0/1189369266 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4be4142c70 msgr2=0x7f4be4138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:49.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.784+0000 7f4be2ffd640 1 --2- 172.21.15.19:0/1189369266 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4be4142c70 0x7f4be4138500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:49.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.784+0000 7f4be2ffd640 10 monclient: _finish_hunting 0 2024-05-04T18:18:49.790 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.784+0000 7f4be2ffd640 1 monclient: found mon.noname-c 2024-05-04T18:18:49.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.784+0000 7f4be2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:49.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.785+0000 7f4be2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T18:18:49.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.785+0000 7f4be2ffd640 1 -- 172.21.15.19:0/1189369266 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be4137520 con 0x7f4be413fd70 2024-05-04T18:18:49.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.787+0000 7f4be2ffd640 1 --2- 172.21.15.19:0/1189369266 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4be413fd70 0x7f4be4137fc0 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0098c0 tx=0x7f4bcc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:49.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.787+0000 7f4bdbfff640 10 client.?.objecter ms_handle_connect 0x7f4be413fd70 2024-05-04T18:18:49.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.787+0000 7f4bdbfff640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:49.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4bdbfff640 1 -- 172.21.15.19:0/1189369266 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc032840 con 0x7f4be413fd70 2024-05-04T18:18:49.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4bdbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:49.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4bdbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T18:18:49.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4bdbfff640 10 monclient: dump: 2024-05-04T18:18:49.794 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:49.794 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:49.794 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.794 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.794 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:49.795 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:49.795 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:49.795 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:49.795 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:49.795 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:49.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4bdbfff640 10 monclient: _finish_auth 0 2024-05-04T18:18:49.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4bdbfff640 10 monclient: _check_auth_tickets 2024-05-04T18:18:49.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4bdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:49.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4bdbfff640 1 -- 172.21.15.19:0/1189369266 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bcc032a00 con 0x7f4be413fd70 2024-05-04T18:18:49.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4bdbfff640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:49.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4bdbfff640 1 -- 172.21.15.19:0/1189369266 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc032df0 con 0x7f4be413fd70 2024-05-04T18:18:49.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4be9a8c640 5 monclient: authenticate success, global_id 18767 2024-05-04T18:18:49.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4be9a8c640 10 monclient: _renew_subs 2024-05-04T18:18:49.796 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4be9a8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:49.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.791+0000 7f4be9a8c640 1 -- 172.21.15.19:0/1189369266 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be4137800 con 0x7f4be413fd70 2024-05-04T18:18:49.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.791+0000 7f4bdbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:49.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.791+0000 7f4bdbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:49.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.790+0000 7f4be1ffb640 4 set_mon_vals no callback set 2024-05-04T18:18:49.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.791+0000 7f4bdbfff640 10 monclient: dump: 2024-05-04T18:18:49.797 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:49.797 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:49.797 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.797 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:49.797 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:49.797 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:49.797 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:49.798 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:49.798 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:49.798 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:49.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.791+0000 7f4be9a8c640 10 client.18767.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:49.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.791+0000 7f4be9a8c640 10 monclient: _renew_subs 2024-05-04T18:18:49.798 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.791+0000 7f4be9a8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:49.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.791+0000 7f4be9a8c640 1 -- 172.21.15.19:0/1189369266 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be4138a40 con 0x7f4be413fd70 2024-05-04T18:18:49.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.791+0000 7f4be9a8c640 1 librados: init done 2024-05-04T18:18:49.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.792+0000 7f4be9a8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:49.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.792+0000 7f4be9a8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:49.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.792+0000 7f4be9a8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:49.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.792+0000 7f4be9a8c640 1 -- 172.21.15.19:0/1189369266 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be4068b10 con 0x7f4be413fd70 2024-05-04T18:18:49.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.811+0000 7f4bdbfff640 1 -- 172.21.15.19:0/1189369266 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4bcc04e430 con 0x7f4be413fd70 2024-05-04T18:18:49.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.811+0000 7f4bdbfff640 1 --2- 172.21.15.19:0/1189369266 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f4bc406d7c0 0x7f4bc406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:49.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.812+0000 7f4be37fe640 1 --2- 172.21.15.19:0/1189369266 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f4bc406d7c0 0x7f4bc406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:49.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.812+0000 7f4be37fe640 10 monclient: get_auth_request con 0x7f4bc406d7c0 auth_method 0 2024-05-04T18:18:49.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.812+0000 7f4be37fe640 1 --2- 172.21.15.19:0/1189369266 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f4bc406d7c0 0x7f4bc406fc80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009b40 tx=0x7f4bd4009190 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:49.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:49.812+0000 7f4bdbfff640 10 client.18767.objecter ms_handle_connect 0x7f4bc406d7c0 2024-05-04T18:18:50.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.082+0000 7f4bdbfff640 1 -- 172.21.15.19:0/1189369266 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f4bcc02f030 con 0x7f4be413fd70 2024-05-04T18:18:50.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.082+0000 7f4bdbfff640 10 client.18767.objecter ms_dispatch 0x7f4be4000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:50.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.082+0000 7f4bdbfff640 3 client.18767.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:50.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.082+0000 7f4bdbfff640 3 client.18767.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:50.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.082+0000 7f4bdbfff640 20 client.18767.objecter dump_active .. 0 homeless 2024-05-04T18:18:50.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.082+0000 7f4bdbfff640 1 -- 172.21.15.19:0/1189369266 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4bcc040070 con 0x7f4be413fd70 2024-05-04T18:18:50.084 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.082+0000 7f4bdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:50.085 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.082+0000 7f4bdbfff640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:18:50.264 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.262+0000 7f4be9a8c640 1 -- 172.21.15.19:0/1189369266 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4be40b3a40 con 0x7f4bc406d7c0 2024-05-04T18:18:50.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.265+0000 7f4bdbfff640 1 -- 172.21.15.19:0/1189369266 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116259 (secure 0 0 0) 0x7f4be40b3a40 con 0x7f4bc406d7c0 2024-05-04T18:18:50.268 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T18:18:50.268 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-05-04T18:18:50.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 10 librados: watch_flush enter 2024-05-04T18:18:50.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 10 librados: watch_flush exit 2024-05-04T18:18:50.270 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 20 client.18767.objecter shutdown clearing up homeless session... 2024-05-04T18:18:50.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 10 client.18767.objecter successfully canceled tick 2024-05-04T18:18:50.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 -- 172.21.15.19:0/1189369266 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f4bc406d7c0 msgr2=0x7f4bc406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:50.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 --2- 172.21.15.19:0/1189369266 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f4bc406d7c0 0x7f4bc406fc80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009b40 tx=0x7f4bd4009190 comp rx=0 tx=0).stop 2024-05-04T18:18:50.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 10 monclient: shutdown 2024-05-04T18:18:50.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:50.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 -- 172.21.15.19:0/1189369266 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4be413fd70 msgr2=0x7f4be4137fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:50.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 --2- 172.21.15.19:0/1189369266 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4be413fd70 0x7f4be4137fc0 secure :-1 s=READY pgs=3162 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0098c0 tx=0x7f4bcc0046c0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 -- 172.21.15.19:0/1189369266 shutdown_connections 2024-05-04T18:18:50.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 --2- 172.21.15.19:0/1189369266 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f4bc406d7c0 0x7f4bc406fc80 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 --2- 172.21.15.19:0/1189369266 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f4be4142c70 0x7f4be4138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 --2- 172.21.15.19:0/1189369266 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f4be413fd70 0x7f4be4137fc0 unknown :-1 s=CLOSED pgs=3162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.271 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 --2- 172.21.15.19:0/1189369266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4be40b2690 0x7f4be4142060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 -- 172.21.15.19:0/1189369266 >> 172.21.15.19:0/1189369266 conn(0x7f4be40b5840 msgr2=0x7f4be40b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:50.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 -- 172.21.15.19:0/1189369266 shutdown_connections 2024-05-04T18:18:50.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 -- 172.21.15.19:0/1189369266 wait complete. 2024-05-04T18:18:50.272 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.268+0000 7f4be9a8c640 1 librados: shutdown 2024-05-04T18:18:50.281 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":2204,"stamp":"2024-05-04T18:18:49.526620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":542360,"kb_used_data":22300,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754432360,"statfs":{"total":773094113280,"available":772538736640,"internally_reserved":0,"allocated":22835200,"data_stored":8395419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":529714736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":175,"apply_latency_ms":175,"commit_latency_ns":175000000,"apply_latency_ns":175000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001257"},"pg_stats":[{"pgid":"14.1e","version":"0'0","reported_seq":2262,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.646924+0000","last_change":"2024-05-04T17:46:46.574386+0000","last_active":"2024-05-04T18:18:47.646924+0000","last_peered":"2024-05-04T18:18:47.646924+0000","last_clean":"2024-05-04T18:18:47.646924+0000","last_became_active":"2024-05-04T17:46:46.518637+0000","last_became_peered":"2024-05-04T17:46:46.518637+0000","last_unstale":"2024-05-04T18:18:47.646924+0000","last_undegraded":"2024-05-04T18:18:47.646924+0000","last_fullsized":"2024-05-04T18:18:47.646924+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:40:26.227815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":2263,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.453243+0000","last_change":"2024-05-04T18:00:26.429131+0000","last_active":"2024-05-04T18:18:47.453243+0000","last_peered":"2024-05-04T18:18:47.453243+0000","last_clean":"2024-05-04T18:18:47.453243+0000","last_became_active":"2024-05-04T18:00:26.421590+0000","last_became_peered":"2024-05-04T18:00:26.421590+0000","last_unstale":"2024-05-04T18:18:47.453243+0000","last_undegraded":"2024-05-04T18:18:47.453243+0000","last_fullsized":"2024-05-04T18:18:47.453243+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:10:52.211055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":2267,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.591503+0000","last_change":"2024-05-04T17:47:40.091354+0000","last_active":"2024-05-04T18:18:47.591503+0000","last_peered":"2024-05-04T18:18:47.591503+0000","last_clean":"2024-05-04T18:18:47.591503+0000","last_became_active":"2024-05-04T17:47:40.078079+0000","last_became_peered":"2024-05-04T17:47:40.078079+0000","last_unstale":"2024-05-04T18:18:47.591503+0000","last_undegraded":"2024-05-04T18:18:47.591503+0000","last_fullsized":"2024-05-04T18:18:47.591503+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:08:09.078519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":2278,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.250747+0000","last_change":"2024-05-04T17:46:53.620558+0000","last_active":"2024-05-04T18:17:25.250747+0000","last_peered":"2024-05-04T18:17:25.250747+0000","last_clean":"2024-05-04T18:17:25.250747+0000","last_became_active":"2024-05-04T17:46:53.602969+0000","last_became_peered":"2024-05-04T17:46:53.602969+0000","last_unstale":"2024-05-04T18:17:25.250747+0000","last_undegraded":"2024-05-04T18:17:25.250747+0000","last_fullsized":"2024-05-04T18:17:25.250747+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:00:17.742135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":2265,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.553741+0000","last_change":"2024-05-04T18:00:25.071880+0000","last_active":"2024-05-04T18:18:47.553741+0000","last_peered":"2024-05-04T18:18:47.553741+0000","last_clean":"2024-05-04T18:18:47.553741+0000","last_became_active":"2024-05-04T18:00:25.064688+0000","last_became_peered":"2024-05-04T18:00:25.064688+0000","last_unstale":"2024-05-04T18:18:47.553741+0000","last_undegraded":"2024-05-04T18:18:47.553741+0000","last_fullsized":"2024-05-04T18:18:47.553741+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:58:15.875261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":2281,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.261803+0000","last_change":"2024-05-04T17:46:53.705592+0000","last_active":"2024-05-04T18:17:25.261803+0000","last_peered":"2024-05-04T18:17:25.261803+0000","last_clean":"2024-05-04T18:17:25.261803+0000","last_became_active":"2024-05-04T17:46:53.682041+0000","last_became_peered":"2024-05-04T17:46:53.682041+0000","last_unstale":"2024-05-04T18:17:25.261803+0000","last_undegraded":"2024-05-04T18:17:25.261803+0000","last_fullsized":"2024-05-04T18:17:25.261803+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:09:58.581895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":2287,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.236439+0000","last_change":"2024-05-04T17:46:35.450534+0000","last_active":"2024-05-04T18:17:25.236439+0000","last_peered":"2024-05-04T18:17:25.236439+0000","last_clean":"2024-05-04T18:17:25.236439+0000","last_became_active":"2024-05-04T17:46:35.438066+0000","last_became_peered":"2024-05-04T17:46:35.438066+0000","last_unstale":"2024-05-04T18:17:25.236439+0000","last_undegraded":"2024-05-04T18:17:25.236439+0000","last_fullsized":"2024-05-04T18:17:25.236439+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:28:24.859168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1","version":"105'7","reported_seq":2269,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.435333+0000","last_change":"2024-05-04T17:46:53.697892+0000","last_active":"2024-05-04T18:18:47.435333+0000","last_peered":"2024-05-04T18:18:47.435333+0000","last_clean":"2024-05-04T18:18:47.435333+0000","last_became_active":"2024-05-04T17:46:53.692194+0000","last_became_peered":"2024-05-04T17:46:53.692194+0000","last_unstale":"2024-05-04T18:18:47.435333+0000","last_undegraded":"2024-05-04T18:18:47.435333+0000","last_fullsized":"2024-05-04T18:18:47.435333+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:29:13.161473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":2279,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.321514+0000","last_change":"2024-05-04T17:46:32.428706+0000","last_active":"2024-05-04T18:18:46.321514+0000","last_peered":"2024-05-04T18:18:46.321514+0000","last_clean":"2024-05-04T18:18:46.321514+0000","last_became_active":"2024-05-04T17:46:32.422070+0000","last_became_peered":"2024-05-04T17:46:32.422070+0000","last_unstale":"2024-05-04T18:18:46.321514+0000","last_undegraded":"2024-05-04T18:18:46.321514+0000","last_fullsized":"2024-05-04T18:18:46.321514+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:08:42.163294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":2240,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.624948+0000","last_change":"2024-05-04T17:47:33.488923+0000","last_active":"2024-05-04T18:18:47.624948+0000","last_peered":"2024-05-04T18:18:47.624948+0000","last_clean":"2024-05-04T18:18:47.624948+0000","last_became_active":"2024-05-04T17:47:33.470216+0000","last_became_peered":"2024-05-04T17:47:33.470216+0000","last_unstale":"2024-05-04T18:18:47.624948+0000","last_undegraded":"2024-05-04T18:18:47.624948+0000","last_fullsized":"2024-05-04T18:18:47.624948+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:31:16.602746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":2247,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.572078+0000","last_change":"2024-05-04T17:46:55.297891+0000","last_active":"2024-05-04T18:18:47.572078+0000","last_peered":"2024-05-04T18:18:47.572078+0000","last_clean":"2024-05-04T18:18:47.572078+0000","last_became_active":"2024-05-04T17:46:55.287626+0000","last_became_peered":"2024-05-04T17:46:55.287626+0000","last_unstale":"2024-05-04T18:18:47.572078+0000","last_undegraded":"2024-05-04T18:18:47.572078+0000","last_fullsized":"2024-05-04T18:18:47.572078+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:47:08.351184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":2280,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.317119+0000","last_change":"2024-05-04T17:46:35.598755+0000","last_active":"2024-05-04T18:18:46.317119+0000","last_peered":"2024-05-04T18:18:46.317119+0000","last_clean":"2024-05-04T18:18:46.317119+0000","last_became_active":"2024-05-04T17:46:35.592584+0000","last_became_peered":"2024-05-04T17:46:35.592584+0000","last_unstale":"2024-05-04T18:18:46.317119+0000","last_undegraded":"2024-05-04T18:18:46.317119+0000","last_fullsized":"2024-05-04T18:18:46.317119+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:35:42.158774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":2240,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.627053+0000","last_change":"2024-05-04T17:47:04.885267+0000","last_active":"2024-05-04T18:18:47.627053+0000","last_peered":"2024-05-04T18:18:47.627053+0000","last_clean":"2024-05-04T18:18:47.627053+0000","last_became_active":"2024-05-04T17:47:04.857381+0000","last_became_peered":"2024-05-04T17:47:04.857381+0000","last_unstale":"2024-05-04T18:18:47.627053+0000","last_undegraded":"2024-05-04T18:18:47.627053+0000","last_fullsized":"2024-05-04T18:18:47.627053+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:09:14.931257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"18'76","reported_seq":2279,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.570917+0000","last_change":"2024-05-04T17:45:18.398964+0000","last_active":"2024-05-04T18:18:47.570917+0000","last_peered":"2024-05-04T18:18:47.570917+0000","last_clean":"2024-05-04T18:18:47.570917+0000","last_became_active":"2024-05-04T17:45:17.720848+0000","last_became_peered":"2024-05-04T17:45:17.720848+0000","last_unstale":"2024-05-04T18:18:47.570917+0000","last_undegraded":"2024-05-04T18:18:47.570917+0000","last_fullsized":"2024-05-04T18:18:47.570917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:03.919442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:03.919442+0000","last_clean_scrub_stamp":"2024-05-04T17:45:03.919442+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:32:22.959179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2279,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.578512+0000","last_change":"2024-05-04T17:45:23.593473+0000","last_active":"2024-05-04T18:18:47.578512+0000","last_peered":"2024-05-04T18:18:47.578512+0000","last_clean":"2024-05-04T18:18:47.578512+0000","last_became_active":"2024-05-04T17:45:23.583946+0000","last_became_peered":"2024-05-04T17:45:23.583946+0000","last_unstale":"2024-05-04T18:18:47.578512+0000","last_undegraded":"2024-05-04T18:18:47.578512+0000","last_fullsized":"2024-05-04T18:18:47.578512+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:19:20.455748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":2280,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.379217+0000","last_change":"2024-05-04T17:46:45.398622+0000","last_active":"2024-05-04T18:18:46.379217+0000","last_peered":"2024-05-04T18:18:46.379217+0000","last_clean":"2024-05-04T18:18:46.379217+0000","last_became_active":"2024-05-04T17:46:45.387152+0000","last_became_peered":"2024-05-04T17:46:45.387152+0000","last_unstale":"2024-05-04T18:18:46.379217+0000","last_undegraded":"2024-05-04T18:18:46.379217+0000","last_fullsized":"2024-05-04T18:18:46.379217+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:51:33.866454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2282,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.569454+0000","last_change":"2024-05-04T17:45:23.292542+0000","last_active":"2024-05-04T18:18:47.569454+0000","last_peered":"2024-05-04T18:18:47.569454+0000","last_clean":"2024-05-04T18:18:47.569454+0000","last_became_active":"2024-05-04T17:45:23.262086+0000","last_became_peered":"2024-05-04T17:45:23.262086+0000","last_unstale":"2024-05-04T18:18:47.569454+0000","last_undegraded":"2024-05-04T18:18:47.569454+0000","last_fullsized":"2024-05-04T18:18:47.569454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:16:49.542097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":2265,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.633827+0000","last_change":"2024-05-04T17:47:14.147039+0000","last_active":"2024-05-04T18:18:47.633827+0000","last_peered":"2024-05-04T18:18:47.633827+0000","last_clean":"2024-05-04T18:18:47.633827+0000","last_became_active":"2024-05-04T17:47:14.130381+0000","last_became_peered":"2024-05-04T17:47:14.130381+0000","last_unstale":"2024-05-04T18:18:47.633827+0000","last_undegraded":"2024-05-04T18:18:47.633827+0000","last_fullsized":"2024-05-04T18:18:47.633827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:56:09.055998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2272,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.438925+0000","last_change":"2024-05-04T17:45:23.301096+0000","last_active":"2024-05-04T18:18:47.438925+0000","last_peered":"2024-05-04T18:18:47.438925+0000","last_clean":"2024-05-04T18:18:47.438925+0000","last_became_active":"2024-05-04T17:45:23.265024+0000","last_became_peered":"2024-05-04T17:45:23.265024+0000","last_unstale":"2024-05-04T18:18:47.438925+0000","last_undegraded":"2024-05-04T18:18:47.438925+0000","last_fullsized":"2024-05-04T18:18:47.438925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:48:10.154363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":2243,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.640584+0000","last_change":"2024-05-04T18:00:26.429267+0000","last_active":"2024-05-04T18:18:47.640584+0000","last_peered":"2024-05-04T18:18:47.640584+0000","last_clean":"2024-05-04T18:18:47.640584+0000","last_became_active":"2024-05-04T18:00:26.421083+0000","last_became_peered":"2024-05-04T18:00:26.421083+0000","last_unstale":"2024-05-04T18:18:47.640584+0000","last_undegraded":"2024-05-04T18:18:47.640584+0000","last_fullsized":"2024-05-04T18:18:47.640584+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:26:20.262350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2280,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.595840+0000","last_change":"2024-05-04T17:45:23.491691+0000","last_active":"2024-05-04T18:18:47.595840+0000","last_peered":"2024-05-04T18:18:47.595840+0000","last_clean":"2024-05-04T18:18:47.595840+0000","last_became_active":"2024-05-04T17:45:23.480170+0000","last_became_peered":"2024-05-04T17:45:23.480170+0000","last_unstale":"2024-05-04T18:18:47.595840+0000","last_undegraded":"2024-05-04T18:18:47.595840+0000","last_fullsized":"2024-05-04T18:18:47.595840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:18:07.981076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":2285,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.237462+0000","last_change":"2024-05-04T17:46:35.456577+0000","last_active":"2024-05-04T18:17:25.237462+0000","last_peered":"2024-05-04T18:17:25.237462+0000","last_clean":"2024-05-04T18:17:25.237462+0000","last_became_active":"2024-05-04T17:46:35.445021+0000","last_became_peered":"2024-05-04T17:46:35.445021+0000","last_unstale":"2024-05-04T18:17:25.237462+0000","last_undegraded":"2024-05-04T18:17:25.237462+0000","last_fullsized":"2024-05-04T18:17:25.237462+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:46:31.130508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2290,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.381308+0000","last_change":"2024-05-04T17:45:23.488141+0000","last_active":"2024-05-04T18:18:46.381308+0000","last_peered":"2024-05-04T18:18:46.381308+0000","last_clean":"2024-05-04T18:18:46.381308+0000","last_became_active":"2024-05-04T17:45:23.481564+0000","last_became_peered":"2024-05-04T17:45:23.481564+0000","last_unstale":"2024-05-04T18:18:46.381308+0000","last_undegraded":"2024-05-04T18:18:46.381308+0000","last_fullsized":"2024-05-04T18:18:46.381308+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:23:41.989802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":2243,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.234068+0000","last_change":"2024-05-04T17:46:37.667579+0000","last_active":"2024-05-04T18:17:25.234068+0000","last_peered":"2024-05-04T18:17:25.234068+0000","last_clean":"2024-05-04T18:17:25.234068+0000","last_became_active":"2024-05-04T17:46:37.661672+0000","last_became_peered":"2024-05-04T17:46:37.661672+0000","last_unstale":"2024-05-04T18:17:25.234068+0000","last_undegraded":"2024-05-04T18:17:25.234068+0000","last_fullsized":"2024-05-04T18:17:25.234068+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:58:33.429360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2281,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.629869+0000","last_change":"2024-05-04T17:45:23.379564+0000","last_active":"2024-05-04T18:18:47.629869+0000","last_peered":"2024-05-04T18:18:47.629869+0000","last_clean":"2024-05-04T18:18:47.629869+0000","last_became_active":"2024-05-04T17:45:23.370117+0000","last_became_peered":"2024-05-04T17:45:23.370117+0000","last_unstale":"2024-05-04T18:18:47.629869+0000","last_undegraded":"2024-05-04T18:18:47.629869+0000","last_fullsized":"2024-05-04T18:18:47.629869+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:39:45.488747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":2259,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.560915+0000","last_change":"2024-05-04T17:47:00.057580+0000","last_active":"2024-05-04T18:18:47.560915+0000","last_peered":"2024-05-04T18:18:47.560915+0000","last_clean":"2024-05-04T18:18:47.560915+0000","last_became_active":"2024-05-04T17:47:00.051980+0000","last_became_peered":"2024-05-04T17:47:00.051980+0000","last_unstale":"2024-05-04T18:18:47.560915+0000","last_undegraded":"2024-05-04T18:18:47.560915+0000","last_fullsized":"2024-05-04T18:18:47.560915+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:01:05.388827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2295,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.258047+0000","last_change":"2024-05-04T17:45:23.713140+0000","last_active":"2024-05-04T18:17:25.258047+0000","last_peered":"2024-05-04T18:17:25.258047+0000","last_clean":"2024-05-04T18:17:25.258047+0000","last_became_active":"2024-05-04T17:45:23.686812+0000","last_became_peered":"2024-05-04T17:45:23.686812+0000","last_unstale":"2024-05-04T18:17:25.258047+0000","last_undegraded":"2024-05-04T18:17:25.258047+0000","last_fullsized":"2024-05-04T18:17:25.258047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:56:49.147299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":2269,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.583227+0000","last_change":"2024-05-04T17:47:40.993471+0000","last_active":"2024-05-04T18:18:47.583227+0000","last_peered":"2024-05-04T18:18:47.583227+0000","last_clean":"2024-05-04T18:18:47.583227+0000","last_became_active":"2024-05-04T17:47:40.988027+0000","last_became_peered":"2024-05-04T17:47:40.988027+0000","last_unstale":"2024-05-04T18:18:47.583227+0000","last_undegraded":"2024-05-04T18:18:47.583227+0000","last_fullsized":"2024-05-04T18:18:47.583227+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:41:52.420496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2276,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.650826+0000","last_change":"2024-05-04T17:45:23.672655+0000","last_active":"2024-05-04T18:18:47.650826+0000","last_peered":"2024-05-04T18:18:47.650826+0000","last_clean":"2024-05-04T18:18:47.650826+0000","last_became_active":"2024-05-04T17:45:23.667080+0000","last_became_peered":"2024-05-04T17:45:23.667080+0000","last_unstale":"2024-05-04T18:18:47.650826+0000","last_undegraded":"2024-05-04T18:18:47.650826+0000","last_fullsized":"2024-05-04T18:18:47.650826+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:43:12.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":2264,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.573341+0000","last_change":"2024-05-04T17:46:40.940632+0000","last_active":"2024-05-04T18:18:47.573341+0000","last_peered":"2024-05-04T18:18:47.573341+0000","last_clean":"2024-05-04T18:18:47.573341+0000","last_became_active":"2024-05-04T17:46:40.915492+0000","last_became_peered":"2024-05-04T17:46:40.915492+0000","last_unstale":"2024-05-04T18:18:47.573341+0000","last_undegraded":"2024-05-04T18:18:47.573341+0000","last_fullsized":"2024-05-04T18:18:47.573341+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:43:04.514591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":2272,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.312727+0000","last_change":"2024-05-04T17:46:53.791115+0000","last_active":"2024-05-04T18:18:46.312727+0000","last_peered":"2024-05-04T18:18:46.312727+0000","last_clean":"2024-05-04T18:18:46.312727+0000","last_became_active":"2024-05-04T17:46:53.785221+0000","last_became_peered":"2024-05-04T17:46:53.785221+0000","last_unstale":"2024-05-04T18:18:46.312727+0000","last_undegraded":"2024-05-04T18:18:46.312727+0000","last_fullsized":"2024-05-04T18:18:46.312727+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:28:11.181195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":2274,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.580658+0000","last_change":"2024-05-04T17:46:43.226742+0000","last_active":"2024-05-04T18:18:47.580658+0000","last_peered":"2024-05-04T18:18:47.580658+0000","last_clean":"2024-05-04T18:18:47.580658+0000","last_became_active":"2024-05-04T17:46:43.193128+0000","last_became_peered":"2024-05-04T17:46:43.193128+0000","last_unstale":"2024-05-04T18:18:47.580658+0000","last_undegraded":"2024-05-04T18:18:47.580658+0000","last_fullsized":"2024-05-04T18:18:47.580658+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:52:43.388505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":2283,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.239474+0000","last_change":"2024-05-04T17:46:42.337619+0000","last_active":"2024-05-04T18:17:25.239474+0000","last_peered":"2024-05-04T18:17:25.239474+0000","last_clean":"2024-05-04T18:17:25.239474+0000","last_became_active":"2024-05-04T17:46:42.332321+0000","last_became_peered":"2024-05-04T17:46:42.332321+0000","last_unstale":"2024-05-04T18:17:25.239474+0000","last_undegraded":"2024-05-04T18:17:25.239474+0000","last_fullsized":"2024-05-04T18:17:25.239474+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:45:24.298610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":2262,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.637251+0000","last_change":"2024-05-04T17:47:27.489125+0000","last_active":"2024-05-04T18:18:47.637251+0000","last_peered":"2024-05-04T18:18:47.637251+0000","last_clean":"2024-05-04T18:18:47.637251+0000","last_became_active":"2024-05-04T17:47:27.462872+0000","last_became_peered":"2024-05-04T17:47:27.462872+0000","last_unstale":"2024-05-04T18:18:47.637251+0000","last_undegraded":"2024-05-04T18:18:47.637251+0000","last_fullsized":"2024-05-04T18:18:47.637251+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:31:19.507264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":2241,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.620547+0000","last_change":"2024-05-04T17:47:04.137847+0000","last_active":"2024-05-04T18:18:47.620547+0000","last_peered":"2024-05-04T18:18:47.620547+0000","last_clean":"2024-05-04T18:18:47.620547+0000","last_became_active":"2024-05-04T17:47:04.099185+0000","last_became_peered":"2024-05-04T17:47:04.099185+0000","last_unstale":"2024-05-04T18:18:47.620547+0000","last_undegraded":"2024-05-04T18:18:47.620547+0000","last_fullsized":"2024-05-04T18:18:47.620547+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:52:57.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":2272,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.270653+0000","last_change":"2024-05-04T17:47:15.314211+0000","last_active":"2024-05-04T18:17:25.270653+0000","last_peered":"2024-05-04T18:17:25.270653+0000","last_clean":"2024-05-04T18:17:25.270653+0000","last_became_active":"2024-05-04T17:47:15.307302+0000","last_became_peered":"2024-05-04T17:47:15.307302+0000","last_unstale":"2024-05-04T18:17:25.270653+0000","last_undegraded":"2024-05-04T18:17:25.270653+0000","last_fullsized":"2024-05-04T18:17:25.270653+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:21:22.778712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":2267,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.627221+0000","last_change":"2024-05-04T17:47:40.632890+0000","last_active":"2024-05-04T18:18:47.627221+0000","last_peered":"2024-05-04T18:18:47.627221+0000","last_clean":"2024-05-04T18:18:47.627221+0000","last_became_active":"2024-05-04T17:47:40.627379+0000","last_became_peered":"2024-05-04T17:47:40.627379+0000","last_unstale":"2024-05-04T18:18:47.627221+0000","last_undegraded":"2024-05-04T18:18:47.627221+0000","last_fullsized":"2024-05-04T18:18:47.627221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:24:44.506896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":2241,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.590521+0000","last_change":"2024-05-04T17:47:33.306252+0000","last_active":"2024-05-04T18:18:47.590521+0000","last_peered":"2024-05-04T18:18:47.590521+0000","last_clean":"2024-05-04T18:18:47.590521+0000","last_became_active":"2024-05-04T17:47:33.296115+0000","last_became_peered":"2024-05-04T17:47:33.296115+0000","last_unstale":"2024-05-04T18:18:47.590521+0000","last_undegraded":"2024-05-04T18:18:47.590521+0000","last_fullsized":"2024-05-04T18:18:47.590521+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:49:52.193030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":2275,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.546231+0000","last_change":"2024-05-04T17:46:42.745978+0000","last_active":"2024-05-04T18:18:47.546231+0000","last_peered":"2024-05-04T18:18:47.546231+0000","last_clean":"2024-05-04T18:18:47.546231+0000","last_became_active":"2024-05-04T17:46:42.724604+0000","last_became_peered":"2024-05-04T17:46:42.724604+0000","last_unstale":"2024-05-04T18:18:47.546231+0000","last_undegraded":"2024-05-04T18:18:47.546231+0000","last_fullsized":"2024-05-04T18:18:47.546231+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:52:59.865599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":2279,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.288621+0000","last_change":"2024-05-04T17:46:45.356387+0000","last_active":"2024-05-04T18:18:46.288621+0000","last_peered":"2024-05-04T18:18:46.288621+0000","last_clean":"2024-05-04T18:18:46.288621+0000","last_became_active":"2024-05-04T17:46:45.347368+0000","last_became_peered":"2024-05-04T17:46:45.347368+0000","last_unstale":"2024-05-04T18:18:46.288621+0000","last_undegraded":"2024-05-04T18:18:46.288621+0000","last_fullsized":"2024-05-04T18:18:46.288621+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:10:48.903095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":2268,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.570558+0000","last_change":"2024-05-04T17:47:38.797623+0000","last_active":"2024-05-04T18:18:47.570558+0000","last_peered":"2024-05-04T18:18:47.570558+0000","last_clean":"2024-05-04T18:18:47.570558+0000","last_became_active":"2024-05-04T17:47:38.791956+0000","last_became_peered":"2024-05-04T17:47:38.791956+0000","last_unstale":"2024-05-04T18:18:47.570558+0000","last_undegraded":"2024-05-04T18:18:47.570558+0000","last_fullsized":"2024-05-04T18:18:47.570558+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:47:30.587754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313621,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66692,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305148,"statfs":{"total":96636764160,"available":96568471552,"internally_reserved":0,"allocated":3166208,"data_stored":1353769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[]},{"osd":0,"up_from":20,"seq":85899346321,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66508,"kb_used_data":3152,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305332,"statfs":{"total":96636764160,"available":96568659968,"internally_reserved":0,"allocated":3227648,"data_stored":1224566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66272,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305568,"statfs":{"total":96636764160,"available":96568901632,"internally_reserved":0,"allocated":2985984,"data_stored":1082307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604379030,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66268,"kb_used_data":2912,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305572,"statfs":{"total":96636764160,"available":96568905728,"internally_reserved":0,"allocated":2981888,"data_stored":1082302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":16,"seq":68719477133,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69980,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301860,"statfs":{"total":96636764160,"available":96565104640,"internally_reserved":0,"allocated":2523136,"data_stored":818417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542547,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66064,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305776,"statfs":{"total":96636764160,"available":96569114624,"internally_reserved":0,"allocated":2523136,"data_stored":814326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542543,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69976,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301864,"statfs":{"total":96636764160,"available":96565108736,"internally_reserved":0,"allocated":2519040,"data_stored":814321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575251,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70600,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301240,"statfs":{"total":96636764160,"available":96564469760,"internally_reserved":0,"allocated":2908160,"data_stored":1205411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T18:18:50.283 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-04T18:18:50.283 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T18:18:50.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.382+0000 7fc66734b640 10 monclient: get_monmap_and_config 2024-05-04T18:18:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.382+0000 7fc66734b640 10 monclient: build_initial_monmap 2024-05-04T18:18:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.382+0000 7fc66734b640 10 monclient: monmap: 2024-05-04T18:18:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:50.384204+0000 2024-05-04T18:18:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:50.384204+0000 2024-05-04T18:18:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:50.384 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:50.385 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:50.385 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:50.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.382+0000 7fc66734b640 1 Processor -- start 2024-05-04T18:18:50.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 1 -- start start 2024-05-04T18:18:50.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 10 monclient: init 2024-05-04T18:18:50.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fc66006dc90 0x7fc66013fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 10 monclient(hunting): picked mon.noname-a con 0x7fc66006dc90 addr v2:172.21.15.19:3300/0 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc66006e070 0x7fc660140070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 10 monclient(hunting): picked mon.noname-c con 0x7fc66006e070 addr v2:172.21.15.186:3300/0 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6601460a0 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc66006d060 con 0x7fc6601460a0 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fc660071d90 con 0x7fc66006dc90 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fc660140660 con 0x7fc66006e070 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.383+0000 7fc66734b640 10 monclient(hunting): authenticate will time out at 3352.498463s 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.384+0000 7fc6648e6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc66006e070 0x7fc660140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.384+0000 7fc6650e7640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fc66006dc90 0x7fc66013fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:50.387 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.384+0000 7fc6650e7640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fc66006dc90 0x7fc66013fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:47814/0 (socket says 172.21.15.19:47814) 2024-05-04T18:18:50.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.384+0000 7fc6650e7640 1 -- 172.21.15.19:0/2071102229 learned_addr learned my addr 172.21.15.19:0/2071102229 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:50.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.384+0000 7fc6650e7640 10 monclient(hunting): get_auth_request con 0x7fc66006dc90 auth_method 0 2024-05-04T18:18:50.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.384+0000 7fc6650e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:50.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.384+0000 7fc6650e7640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:50.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.384+0000 7fc6650e7640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:50.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.384+0000 7fc6648e6640 10 monclient(hunting): get_auth_request con 0x7fc66006e070 auth_method 0 2024-05-04T18:18:50.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.384+0000 7fc6648e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:50.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.384+0000 7fc6648e6640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:50.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.384+0000 7fc6648e6640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:50.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.385+0000 7fc6648e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:50.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.385+0000 7fc6648e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:50.388 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.385+0000 7fc6648e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:50.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.386+0000 7fc6650e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:50.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.386+0000 7fc6650e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:50.389 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.386+0000 7fc6650e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:50.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.387+0000 7fc6648e6640 10 monclient(hunting): handle_auth_done global_id 22720 payload 274 2024-05-04T18:18:50.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.387+0000 7fc6648e6640 1 -- 172.21.15.19:0/2071102229 >> v2:172.21.15.19:3300/0 conn(0x7fc66006dc90 msgr2=0x7fc66013fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:50.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.387+0000 7fc6648e6640 1 --2- 172.21.15.19:0/2071102229 >> v2:172.21.15.19:3300/0 conn(0x7fc66006dc90 0x7fc66013fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.387+0000 7fc6648e6640 1 -- 172.21.15.19:0/2071102229 >> v1:172.21.15.19:6789/0 conn(0x7fc6601460a0 legacy=0x7fc6601464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:50.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.387+0000 7fc6648e6640 10 monclient: _finish_hunting 0 2024-05-04T18:18:50.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.387+0000 7fc6648e6640 1 monclient: found mon.noname-c 2024-05-04T18:18:50.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.387+0000 7fc6648e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:50.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.387+0000 7fc6648e6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T18:18:50.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.387+0000 7fc6648e6640 1 -- 172.21.15.19:0/2071102229 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc660073790 con 0x7fc66006e070 2024-05-04T18:18:50.390 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.389+0000 7fc6648e6640 1 --2- 172.21.15.19:0/2071102229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc66006e070 0x7fc660140070 secure :-1 s=READY pgs=5818 cs=0 l=1 rev1=1 crypto rx=0x7fc6600b75f0 tx=0x7fc65002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45ece8e3f1a94c3b server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:50.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 1 -- 172.21.15.19:0/2071102229 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6500315a0 con 0x7fc66006e070 2024-05-04T18:18:50.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:50.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T18:18:50.394 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 10 monclient: dump: 2024-05-04T18:18:50.394 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:50.395 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:50.395 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:50.395 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:50.396 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:50.396 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:50.396 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:50.396 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:50.396 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:50.396 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:50.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 10 monclient: _finish_auth 0 2024-05-04T18:18:50.396 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 10 monclient: _check_auth_tickets 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 1 -- 172.21.15.19:0/2071102229 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc650037070 con 0x7fc66006e070 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 1 -- 172.21.15.19:0/2071102229 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc650031960 con 0x7fc66006e070 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc66734b640 5 monclient: authenticate success, global_id 22720 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc66734b640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc66734b640 4 set_mon_vals no callback set 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc657fff640 10 monclient: dump: 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:50.397 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:50.398 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:50.398 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:50.398 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:50.398 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:50.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc66734b640 10 monclient: shutdown 2024-05-04T18:18:50.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc66734b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:50.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc66734b640 1 -- 172.21.15.19:0/2071102229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc66006e070 msgr2=0x7fc660140070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:50.402 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc66734b640 1 --2- 172.21.15.19:0/2071102229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc66006e070 0x7fc660140070 secure :-1 s=READY pgs=5818 cs=0 l=1 rev1=1 crypto rx=0x7fc6600b75f0 tx=0x7fc65002b840 comp rx=0 tx=0).stop 2024-05-04T18:18:50.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc66734b640 1 -- 172.21.15.19:0/2071102229 shutdown_connections 2024-05-04T18:18:50.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc66734b640 1 --2- 172.21.15.19:0/2071102229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc66006e070 0x7fc660140070 unknown :-1 s=CLOSED pgs=5818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc66734b640 1 --2- 172.21.15.19:0/2071102229 >> v2:172.21.15.19:3300/0 conn(0x7fc66006dc90 0x7fc66013fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.393+0000 7fc66734b640 1 -- 172.21.15.19:0/2071102229 >> 172.21.15.19:0/2071102229 conn(0x7fc6600b6980 msgr2=0x7fc6600b6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:50.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.394+0000 7fc66734b640 1 -- 172.21.15.19:0/2071102229 shutdown_connections 2024-05-04T18:18:50.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.394+0000 7fc66734b640 1 -- 172.21.15.19:0/2071102229 wait complete. 2024-05-04T18:18:50.403 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.394+0000 7fc66734b640 10 monclient: build_initial_monmap 2024-05-04T18:18:50.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.394+0000 7fc66734b640 10 monclient: monmap: 2024-05-04T18:18:50.404 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:50.404 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:50.404 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:50.404 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:50.404 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:50.404 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:50.404 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:50.404 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:50.404 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:50.404 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:50.404 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.394+0000 7fc66734b640 1 librados: starting msgr at 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.394+0000 7fc66734b640 1 librados: starting objecter 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 1 Processor -- start 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 1 -- start start 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 1 librados: setting wanted keys 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 1 librados: calling monclient init 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 10 monclient: init 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc66006dc90 0x7fc660137500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 10 monclient(hunting): picked mon.noname-a con 0x7fc66006dc90 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc66006e070 0x7fc6600af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 10 monclient(hunting): picked mon.noname-b con 0x7fc66006e070 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc6601460a0 0x7fc6601d7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 10 monclient(hunting): picked mon.noname-c con 0x7fc6601460a0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:50.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc66014d5b0 con 0x7fc66006dc90 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc66014d430 con 0x7fc66006e070 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc66014d730 con 0x7fc6601460a0 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.395+0000 7fc66734b640 10 monclient(hunting): authenticate will time out at 3352.510248s 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6650e7640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc66006dc90 0x7fc660137500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6648e6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc66006e070 0x7fc6600af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6658e8640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc6601460a0 0x7fc6601d7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6650e7640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc66006dc90 0x7fc660137500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:47820/0 (socket says 172.21.15.19:47820) 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6650e7640 1 -- 172.21.15.19:0/3143748513 learned_addr learned my addr 172.21.15.19:0/3143748513 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6650e7640 10 monclient(hunting): get_auth_request con 0x7fc66006dc90 auth_method 0 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6648e6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc66006e070 0x7fc6600af3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:43616/0 (socket says 172.21.15.19:43616) 2024-05-04T18:18:50.406 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6650e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6650e7640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6650e7640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6648e6640 10 monclient(hunting): get_auth_request con 0x7fc66006e070 auth_method 0 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6648e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6648e6640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6648e6640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6658e8640 10 monclient(hunting): get_auth_request con 0x7fc6601460a0 auth_method 0 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6658e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6658e8640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.396+0000 7fc6658e8640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.397+0000 7fc6650e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.397+0000 7fc6650e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.397+0000 7fc6650e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.397+0000 7fc6648e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:50.407 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.397+0000 7fc6648e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.397+0000 7fc6648e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.397+0000 7fc6658e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.397+0000 7fc6658e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.397+0000 7fc6658e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.400+0000 7fc6648e6640 10 monclient(hunting): handle_auth_done global_id 22723 payload 824 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.400+0000 7fc6648e6640 1 -- 172.21.15.19:0/3143748513 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc6601460a0 msgr2=0x7fc6601d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.400+0000 7fc6648e6640 1 --2- 172.21.15.19:0/3143748513 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc6601460a0 0x7fc6601d7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.400+0000 7fc6648e6640 1 -- 172.21.15.19:0/3143748513 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc66006dc90 msgr2=0x7fc660137500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.400+0000 7fc6648e6640 1 --2- 172.21.15.19:0/3143748513 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc66006dc90 0x7fc660137500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.400+0000 7fc6648e6640 10 monclient: _finish_hunting 0 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.400+0000 7fc6648e6640 1 monclient: found mon.noname-b 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.400+0000 7fc6648e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.400+0000 7fc6648e6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:50.408 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.400+0000 7fc6648e6640 1 -- 172.21.15.19:0/3143748513 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc660141640 con 0x7fc66006e070 2024-05-04T18:18:50.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.402+0000 7fc6648e6640 1 --2- 172.21.15.19:0/3143748513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc66006e070 0x7fc6600af3a0 secure :-1 s=READY pgs=5819 cs=0 l=1 rev1=1 crypto rx=0x7fc650033000 tx=0x7fc6500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:50.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.402+0000 7fc655ffb640 10 client.?.objecter ms_handle_connect 0x7fc66006e070 2024-05-04T18:18:50.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.402+0000 7fc655ffb640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:50.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 1 -- 172.21.15.19:0/3143748513 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc650032630 con 0x7fc66006e070 2024-05-04T18:18:50.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:50.410 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:50.411 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 10 monclient: dump: 2024-05-04T18:18:50.411 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:50.411 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:50.411 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:50.411 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:50.411 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:50.412 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:50.412 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:50.412 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:50.412 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:50.412 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:50.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 10 monclient: _finish_auth 0 2024-05-04T18:18:50.412 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 10 monclient: _check_auth_tickets 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 1 -- 172.21.15.19:0/3143748513 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6500327d0 con 0x7fc66006e070 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc66734b640 5 monclient: authenticate success, global_id 22723 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc66734b640 10 monclient: _renew_subs 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc66734b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc66734b640 1 -- 172.21.15.19:0/3143748513 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6600af8e0 con 0x7fc66006e070 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 1 -- 172.21.15.19:0/3143748513 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc650032ba0 con 0x7fc66006e070 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc655ffb640 10 monclient: dump: 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:50.413 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc656ffd640 4 set_mon_vals no callback set 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc66734b640 10 client.22723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc66734b640 10 monclient: _renew_subs 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc66734b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:50.414 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc66734b640 1 -- 172.21.15.19:0/3143748513 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6600b0450 con 0x7fc66006e070 2024-05-04T18:18:50.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.406+0000 7fc66734b640 1 librados: init done 2024-05-04T18:18:50.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.407+0000 7fc66734b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:50.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.407+0000 7fc66734b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:50.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.407+0000 7fc66734b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:50.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.407+0000 7fc66734b640 1 -- 172.21.15.19:0/3143748513 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc660068b10 con 0x7fc66006e070 2024-05-04T18:18:50.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.427+0000 7fc655ffb640 1 -- 172.21.15.19:0/3143748513 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc65004d3f0 con 0x7fc66006e070 2024-05-04T18:18:50.428 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.427+0000 7fc655ffb640 1 --2- 172.21.15.19:0/3143748513 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc64406d810 0x7fc64406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:50.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.427+0000 7fc6650e7640 1 --2- 172.21.15.19:0/3143748513 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc64406d810 0x7fc64406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:50.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.427+0000 7fc6650e7640 10 monclient: get_auth_request con 0x7fc64406d810 auth_method 0 2024-05-04T18:18:50.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.428+0000 7fc6650e7640 1 --2- 172.21.15.19:0/3143748513 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc64406d810 0x7fc64406fcd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0094a0 tx=0x7fc65c009240 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:50.430 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.428+0000 7fc655ffb640 10 client.22723.objecter ms_handle_connect 0x7fc64406d810 2024-05-04T18:18:50.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.661+0000 7fc655ffb640 1 -- 172.21.15.19:0/3143748513 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7fc65002f020 con 0x7fc66006e070 2024-05-04T18:18:50.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.662+0000 7fc655ffb640 10 client.22723.objecter ms_dispatch 0x7fc660000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:50.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.662+0000 7fc655ffb640 3 client.22723.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:50.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.662+0000 7fc655ffb640 3 client.22723.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:50.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.662+0000 7fc655ffb640 20 client.22723.objecter dump_active .. 0 homeless 2024-05-04T18:18:50.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.672+0000 7fc655ffb640 1 -- 172.21.15.19:0/3143748513 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc650040070 con 0x7fc66006e070 2024-05-04T18:18:50.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.672+0000 7fc655ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:50.673 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.672+0000 7fc655ffb640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:18:50.874 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.873+0000 7fc66734b640 1 -- 172.21.15.19:0/3143748513 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc66006b9f0 con 0x7fc64406d810 2024-05-04T18:18:50.877 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.876+0000 7fc655ffb640 1 -- 172.21.15.19:0/3143748513 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116259 (secure 0 0 0) 0x7fc66006b9f0 con 0x7fc64406d810 2024-05-04T18:18:50.878 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T18:18:50.879 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-05-04T18:18:50.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.878+0000 7fc66734b640 10 librados: watch_flush enter 2024-05-04T18:18:50.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.878+0000 7fc66734b640 10 librados: watch_flush exit 2024-05-04T18:18:50.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.878+0000 7fc66734b640 20 client.22723.objecter shutdown clearing up homeless session... 2024-05-04T18:18:50.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.878+0000 7fc66734b640 10 client.22723.objecter successfully canceled tick 2024-05-04T18:18:50.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 -- 172.21.15.19:0/3143748513 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc64406d810 msgr2=0x7fc64406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:50.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 --2- 172.21.15.19:0/3143748513 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc64406d810 0x7fc64406fcd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc65c0094a0 tx=0x7fc65c009240 comp rx=0 tx=0).stop 2024-05-04T18:18:50.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 10 monclient: shutdown 2024-05-04T18:18:50.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:50.881 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 -- 172.21.15.19:0/3143748513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc66006e070 msgr2=0x7fc6600af3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:50.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 --2- 172.21.15.19:0/3143748513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc66006e070 0x7fc6600af3a0 secure :-1 s=READY pgs=5819 cs=0 l=1 rev1=1 crypto rx=0x7fc650033000 tx=0x7fc6500046c0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 -- 172.21.15.19:0/3143748513 shutdown_connections 2024-05-04T18:18:50.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 --2- 172.21.15.19:0/3143748513 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc64406d810 0x7fc64406fcd0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 --2- 172.21.15.19:0/3143748513 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc6601460a0 0x7fc6601d7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 --2- 172.21.15.19:0/3143748513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc66006e070 0x7fc6600af3a0 unknown :-1 s=CLOSED pgs=5819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 --2- 172.21.15.19:0/3143748513 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc66006dc90 0x7fc660137500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:50.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 -- 172.21.15.19:0/3143748513 >> 172.21.15.19:0/3143748513 conn(0x7fc6600b6980 msgr2=0x7fc6600b53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:50.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 -- 172.21.15.19:0/3143748513 shutdown_connections 2024-05-04T18:18:50.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 -- 172.21.15.19:0/3143748513 wait complete. 2024-05-04T18:18:50.882 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.879+0000 7fc66734b640 1 librados: shutdown 2024-05-04T18:18:50.891 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":2204,"stamp":"2024-05-04T18:18:49.526620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":542360,"kb_used_data":22300,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754432360,"statfs":{"total":773094113280,"available":772538736640,"internally_reserved":0,"allocated":22835200,"data_stored":8395419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":529714736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":175,"apply_latency_ms":175,"commit_latency_ns":175000000,"apply_latency_ns":175000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001257"},"pg_stats":[{"pgid":"14.1e","version":"0'0","reported_seq":2262,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.646924+0000","last_change":"2024-05-04T17:46:46.574386+0000","last_active":"2024-05-04T18:18:47.646924+0000","last_peered":"2024-05-04T18:18:47.646924+0000","last_clean":"2024-05-04T18:18:47.646924+0000","last_became_active":"2024-05-04T17:46:46.518637+0000","last_became_peered":"2024-05-04T17:46:46.518637+0000","last_unstale":"2024-05-04T18:18:47.646924+0000","last_undegraded":"2024-05-04T18:18:47.646924+0000","last_fullsized":"2024-05-04T18:18:47.646924+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:40:26.227815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":2263,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.453243+0000","last_change":"2024-05-04T18:00:26.429131+0000","last_active":"2024-05-04T18:18:47.453243+0000","last_peered":"2024-05-04T18:18:47.453243+0000","last_clean":"2024-05-04T18:18:47.453243+0000","last_became_active":"2024-05-04T18:00:26.421590+0000","last_became_peered":"2024-05-04T18:00:26.421590+0000","last_unstale":"2024-05-04T18:18:47.453243+0000","last_undegraded":"2024-05-04T18:18:47.453243+0000","last_fullsized":"2024-05-04T18:18:47.453243+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:10:52.211055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":2267,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.591503+0000","last_change":"2024-05-04T17:47:40.091354+0000","last_active":"2024-05-04T18:18:47.591503+0000","last_peered":"2024-05-04T18:18:47.591503+0000","last_clean":"2024-05-04T18:18:47.591503+0000","last_became_active":"2024-05-04T17:47:40.078079+0000","last_became_peered":"2024-05-04T17:47:40.078079+0000","last_unstale":"2024-05-04T18:18:47.591503+0000","last_undegraded":"2024-05-04T18:18:47.591503+0000","last_fullsized":"2024-05-04T18:18:47.591503+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:08:09.078519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":2278,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.250747+0000","last_change":"2024-05-04T17:46:53.620558+0000","last_active":"2024-05-04T18:17:25.250747+0000","last_peered":"2024-05-04T18:17:25.250747+0000","last_clean":"2024-05-04T18:17:25.250747+0000","last_became_active":"2024-05-04T17:46:53.602969+0000","last_became_peered":"2024-05-04T17:46:53.602969+0000","last_unstale":"2024-05-04T18:17:25.250747+0000","last_undegraded":"2024-05-04T18:17:25.250747+0000","last_fullsized":"2024-05-04T18:17:25.250747+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:00:17.742135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":2265,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.553741+0000","last_change":"2024-05-04T18:00:25.071880+0000","last_active":"2024-05-04T18:18:47.553741+0000","last_peered":"2024-05-04T18:18:47.553741+0000","last_clean":"2024-05-04T18:18:47.553741+0000","last_became_active":"2024-05-04T18:00:25.064688+0000","last_became_peered":"2024-05-04T18:00:25.064688+0000","last_unstale":"2024-05-04T18:18:47.553741+0000","last_undegraded":"2024-05-04T18:18:47.553741+0000","last_fullsized":"2024-05-04T18:18:47.553741+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:58:15.875261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":2281,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.261803+0000","last_change":"2024-05-04T17:46:53.705592+0000","last_active":"2024-05-04T18:17:25.261803+0000","last_peered":"2024-05-04T18:17:25.261803+0000","last_clean":"2024-05-04T18:17:25.261803+0000","last_became_active":"2024-05-04T17:46:53.682041+0000","last_became_peered":"2024-05-04T17:46:53.682041+0000","last_unstale":"2024-05-04T18:17:25.261803+0000","last_undegraded":"2024-05-04T18:17:25.261803+0000","last_fullsized":"2024-05-04T18:17:25.261803+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:09:58.581895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":2287,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.236439+0000","last_change":"2024-05-04T17:46:35.450534+0000","last_active":"2024-05-04T18:17:25.236439+0000","last_peered":"2024-05-04T18:17:25.236439+0000","last_clean":"2024-05-04T18:17:25.236439+0000","last_became_active":"2024-05-04T17:46:35.438066+0000","last_became_peered":"2024-05-04T17:46:35.438066+0000","last_unstale":"2024-05-04T18:17:25.236439+0000","last_undegraded":"2024-05-04T18:17:25.236439+0000","last_fullsized":"2024-05-04T18:17:25.236439+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:28:24.859168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1","version":"105'7","reported_seq":2269,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.435333+0000","last_change":"2024-05-04T17:46:53.697892+0000","last_active":"2024-05-04T18:18:47.435333+0000","last_peered":"2024-05-04T18:18:47.435333+0000","last_clean":"2024-05-04T18:18:47.435333+0000","last_became_active":"2024-05-04T17:46:53.692194+0000","last_became_peered":"2024-05-04T17:46:53.692194+0000","last_unstale":"2024-05-04T18:18:47.435333+0000","last_undegraded":"2024-05-04T18:18:47.435333+0000","last_fullsized":"2024-05-04T18:18:47.435333+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:29:13.161473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":2279,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.321514+0000","last_change":"2024-05-04T17:46:32.428706+0000","last_active":"2024-05-04T18:18:46.321514+0000","last_peered":"2024-05-04T18:18:46.321514+0000","last_clean":"2024-05-04T18:18:46.321514+0000","last_became_active":"2024-05-04T17:46:32.422070+0000","last_became_peered":"2024-05-04T17:46:32.422070+0000","last_unstale":"2024-05-04T18:18:46.321514+0000","last_undegraded":"2024-05-04T18:18:46.321514+0000","last_fullsized":"2024-05-04T18:18:46.321514+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:08:42.163294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":2240,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.624948+0000","last_change":"2024-05-04T17:47:33.488923+0000","last_active":"2024-05-04T18:18:47.624948+0000","last_peered":"2024-05-04T18:18:47.624948+0000","last_clean":"2024-05-04T18:18:47.624948+0000","last_became_active":"2024-05-04T17:47:33.470216+0000","last_became_peered":"2024-05-04T17:47:33.470216+0000","last_unstale":"2024-05-04T18:18:47.624948+0000","last_undegraded":"2024-05-04T18:18:47.624948+0000","last_fullsized":"2024-05-04T18:18:47.624948+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:31:16.602746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":2247,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.572078+0000","last_change":"2024-05-04T17:46:55.297891+0000","last_active":"2024-05-04T18:18:47.572078+0000","last_peered":"2024-05-04T18:18:47.572078+0000","last_clean":"2024-05-04T18:18:47.572078+0000","last_became_active":"2024-05-04T17:46:55.287626+0000","last_became_peered":"2024-05-04T17:46:55.287626+0000","last_unstale":"2024-05-04T18:18:47.572078+0000","last_undegraded":"2024-05-04T18:18:47.572078+0000","last_fullsized":"2024-05-04T18:18:47.572078+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:47:08.351184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":2280,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.317119+0000","last_change":"2024-05-04T17:46:35.598755+0000","last_active":"2024-05-04T18:18:46.317119+0000","last_peered":"2024-05-04T18:18:46.317119+0000","last_clean":"2024-05-04T18:18:46.317119+0000","last_became_active":"2024-05-04T17:46:35.592584+0000","last_became_peered":"2024-05-04T17:46:35.592584+0000","last_unstale":"2024-05-04T18:18:46.317119+0000","last_undegraded":"2024-05-04T18:18:46.317119+0000","last_fullsized":"2024-05-04T18:18:46.317119+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:35:42.158774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":2240,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.627053+0000","last_change":"2024-05-04T17:47:04.885267+0000","last_active":"2024-05-04T18:18:47.627053+0000","last_peered":"2024-05-04T18:18:47.627053+0000","last_clean":"2024-05-04T18:18:47.627053+0000","last_became_active":"2024-05-04T17:47:04.857381+0000","last_became_peered":"2024-05-04T17:47:04.857381+0000","last_unstale":"2024-05-04T18:18:47.627053+0000","last_undegraded":"2024-05-04T18:18:47.627053+0000","last_fullsized":"2024-05-04T18:18:47.627053+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:09:14.931257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"18'76","reported_seq":2279,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.570917+0000","last_change":"2024-05-04T17:45:18.398964+0000","last_active":"2024-05-04T18:18:47.570917+0000","last_peered":"2024-05-04T18:18:47.570917+0000","last_clean":"2024-05-04T18:18:47.570917+0000","last_became_active":"2024-05-04T17:45:17.720848+0000","last_became_peered":"2024-05-04T17:45:17.720848+0000","last_unstale":"2024-05-04T18:18:47.570917+0000","last_undegraded":"2024-05-04T18:18:47.570917+0000","last_fullsized":"2024-05-04T18:18:47.570917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:03.919442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:03.919442+0000","last_clean_scrub_stamp":"2024-05-04T17:45:03.919442+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:32:22.959179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2279,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.578512+0000","last_change":"2024-05-04T17:45:23.593473+0000","last_active":"2024-05-04T18:18:47.578512+0000","last_peered":"2024-05-04T18:18:47.578512+0000","last_clean":"2024-05-04T18:18:47.578512+0000","last_became_active":"2024-05-04T17:45:23.583946+0000","last_became_peered":"2024-05-04T17:45:23.583946+0000","last_unstale":"2024-05-04T18:18:47.578512+0000","last_undegraded":"2024-05-04T18:18:47.578512+0000","last_fullsized":"2024-05-04T18:18:47.578512+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:19:20.455748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":2280,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.379217+0000","last_change":"2024-05-04T17:46:45.398622+0000","last_active":"2024-05-04T18:18:46.379217+0000","last_peered":"2024-05-04T18:18:46.379217+0000","last_clean":"2024-05-04T18:18:46.379217+0000","last_became_active":"2024-05-04T17:46:45.387152+0000","last_became_peered":"2024-05-04T17:46:45.387152+0000","last_unstale":"2024-05-04T18:18:46.379217+0000","last_undegraded":"2024-05-04T18:18:46.379217+0000","last_fullsized":"2024-05-04T18:18:46.379217+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:51:33.866454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2282,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.569454+0000","last_change":"2024-05-04T17:45:23.292542+0000","last_active":"2024-05-04T18:18:47.569454+0000","last_peered":"2024-05-04T18:18:47.569454+0000","last_clean":"2024-05-04T18:18:47.569454+0000","last_became_active":"2024-05-04T17:45:23.262086+0000","last_became_peered":"2024-05-04T17:45:23.262086+0000","last_unstale":"2024-05-04T18:18:47.569454+0000","last_undegraded":"2024-05-04T18:18:47.569454+0000","last_fullsized":"2024-05-04T18:18:47.569454+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:16:49.542097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":2265,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.633827+0000","last_change":"2024-05-04T17:47:14.147039+0000","last_active":"2024-05-04T18:18:47.633827+0000","last_peered":"2024-05-04T18:18:47.633827+0000","last_clean":"2024-05-04T18:18:47.633827+0000","last_became_active":"2024-05-04T17:47:14.130381+0000","last_became_peered":"2024-05-04T17:47:14.130381+0000","last_unstale":"2024-05-04T18:18:47.633827+0000","last_undegraded":"2024-05-04T18:18:47.633827+0000","last_fullsized":"2024-05-04T18:18:47.633827+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:56:09.055998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2272,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.438925+0000","last_change":"2024-05-04T17:45:23.301096+0000","last_active":"2024-05-04T18:18:47.438925+0000","last_peered":"2024-05-04T18:18:47.438925+0000","last_clean":"2024-05-04T18:18:47.438925+0000","last_became_active":"2024-05-04T17:45:23.265024+0000","last_became_peered":"2024-05-04T17:45:23.265024+0000","last_unstale":"2024-05-04T18:18:47.438925+0000","last_undegraded":"2024-05-04T18:18:47.438925+0000","last_fullsized":"2024-05-04T18:18:47.438925+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:48:10.154363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":2243,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.640584+0000","last_change":"2024-05-04T18:00:26.429267+0000","last_active":"2024-05-04T18:18:47.640584+0000","last_peered":"2024-05-04T18:18:47.640584+0000","last_clean":"2024-05-04T18:18:47.640584+0000","last_became_active":"2024-05-04T18:00:26.421083+0000","last_became_peered":"2024-05-04T18:00:26.421083+0000","last_unstale":"2024-05-04T18:18:47.640584+0000","last_undegraded":"2024-05-04T18:18:47.640584+0000","last_fullsized":"2024-05-04T18:18:47.640584+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:26:20.262350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2280,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.595840+0000","last_change":"2024-05-04T17:45:23.491691+0000","last_active":"2024-05-04T18:18:47.595840+0000","last_peered":"2024-05-04T18:18:47.595840+0000","last_clean":"2024-05-04T18:18:47.595840+0000","last_became_active":"2024-05-04T17:45:23.480170+0000","last_became_peered":"2024-05-04T17:45:23.480170+0000","last_unstale":"2024-05-04T18:18:47.595840+0000","last_undegraded":"2024-05-04T18:18:47.595840+0000","last_fullsized":"2024-05-04T18:18:47.595840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:18:07.981076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":2285,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.237462+0000","last_change":"2024-05-04T17:46:35.456577+0000","last_active":"2024-05-04T18:17:25.237462+0000","last_peered":"2024-05-04T18:17:25.237462+0000","last_clean":"2024-05-04T18:17:25.237462+0000","last_became_active":"2024-05-04T17:46:35.445021+0000","last_became_peered":"2024-05-04T17:46:35.445021+0000","last_unstale":"2024-05-04T18:17:25.237462+0000","last_undegraded":"2024-05-04T18:17:25.237462+0000","last_fullsized":"2024-05-04T18:17:25.237462+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:46:31.130508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2290,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.381308+0000","last_change":"2024-05-04T17:45:23.488141+0000","last_active":"2024-05-04T18:18:46.381308+0000","last_peered":"2024-05-04T18:18:46.381308+0000","last_clean":"2024-05-04T18:18:46.381308+0000","last_became_active":"2024-05-04T17:45:23.481564+0000","last_became_peered":"2024-05-04T17:45:23.481564+0000","last_unstale":"2024-05-04T18:18:46.381308+0000","last_undegraded":"2024-05-04T18:18:46.381308+0000","last_fullsized":"2024-05-04T18:18:46.381308+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:23:41.989802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":2243,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.234068+0000","last_change":"2024-05-04T17:46:37.667579+0000","last_active":"2024-05-04T18:17:25.234068+0000","last_peered":"2024-05-04T18:17:25.234068+0000","last_clean":"2024-05-04T18:17:25.234068+0000","last_became_active":"2024-05-04T17:46:37.661672+0000","last_became_peered":"2024-05-04T17:46:37.661672+0000","last_unstale":"2024-05-04T18:17:25.234068+0000","last_undegraded":"2024-05-04T18:17:25.234068+0000","last_fullsized":"2024-05-04T18:17:25.234068+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:58:33.429360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2281,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.629869+0000","last_change":"2024-05-04T17:45:23.379564+0000","last_active":"2024-05-04T18:18:47.629869+0000","last_peered":"2024-05-04T18:18:47.629869+0000","last_clean":"2024-05-04T18:18:47.629869+0000","last_became_active":"2024-05-04T17:45:23.370117+0000","last_became_peered":"2024-05-04T17:45:23.370117+0000","last_unstale":"2024-05-04T18:18:47.629869+0000","last_undegraded":"2024-05-04T18:18:47.629869+0000","last_fullsized":"2024-05-04T18:18:47.629869+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:39:45.488747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":2259,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.560915+0000","last_change":"2024-05-04T17:47:00.057580+0000","last_active":"2024-05-04T18:18:47.560915+0000","last_peered":"2024-05-04T18:18:47.560915+0000","last_clean":"2024-05-04T18:18:47.560915+0000","last_became_active":"2024-05-04T17:47:00.051980+0000","last_became_peered":"2024-05-04T17:47:00.051980+0000","last_unstale":"2024-05-04T18:18:47.560915+0000","last_undegraded":"2024-05-04T18:18:47.560915+0000","last_fullsized":"2024-05-04T18:18:47.560915+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:01:05.388827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2295,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.258047+0000","last_change":"2024-05-04T17:45:23.713140+0000","last_active":"2024-05-04T18:17:25.258047+0000","last_peered":"2024-05-04T18:17:25.258047+0000","last_clean":"2024-05-04T18:17:25.258047+0000","last_became_active":"2024-05-04T17:45:23.686812+0000","last_became_peered":"2024-05-04T17:45:23.686812+0000","last_unstale":"2024-05-04T18:17:25.258047+0000","last_undegraded":"2024-05-04T18:17:25.258047+0000","last_fullsized":"2024-05-04T18:17:25.258047+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:56:49.147299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":2269,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.583227+0000","last_change":"2024-05-04T17:47:40.993471+0000","last_active":"2024-05-04T18:18:47.583227+0000","last_peered":"2024-05-04T18:18:47.583227+0000","last_clean":"2024-05-04T18:18:47.583227+0000","last_became_active":"2024-05-04T17:47:40.988027+0000","last_became_peered":"2024-05-04T17:47:40.988027+0000","last_unstale":"2024-05-04T18:18:47.583227+0000","last_undegraded":"2024-05-04T18:18:47.583227+0000","last_fullsized":"2024-05-04T18:18:47.583227+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:41:52.420496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2276,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.650826+0000","last_change":"2024-05-04T17:45:23.672655+0000","last_active":"2024-05-04T18:18:47.650826+0000","last_peered":"2024-05-04T18:18:47.650826+0000","last_clean":"2024-05-04T18:18:47.650826+0000","last_became_active":"2024-05-04T17:45:23.667080+0000","last_became_peered":"2024-05-04T17:45:23.667080+0000","last_unstale":"2024-05-04T18:18:47.650826+0000","last_undegraded":"2024-05-04T18:18:47.650826+0000","last_fullsized":"2024-05-04T18:18:47.650826+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:43:12.472773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":2264,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.573341+0000","last_change":"2024-05-04T17:46:40.940632+0000","last_active":"2024-05-04T18:18:47.573341+0000","last_peered":"2024-05-04T18:18:47.573341+0000","last_clean":"2024-05-04T18:18:47.573341+0000","last_became_active":"2024-05-04T17:46:40.915492+0000","last_became_peered":"2024-05-04T17:46:40.915492+0000","last_unstale":"2024-05-04T18:18:47.573341+0000","last_undegraded":"2024-05-04T18:18:47.573341+0000","last_fullsized":"2024-05-04T18:18:47.573341+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:43:04.514591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":2272,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.312727+0000","last_change":"2024-05-04T17:46:53.791115+0000","last_active":"2024-05-04T18:18:46.312727+0000","last_peered":"2024-05-04T18:18:46.312727+0000","last_clean":"2024-05-04T18:18:46.312727+0000","last_became_active":"2024-05-04T17:46:53.785221+0000","last_became_peered":"2024-05-04T17:46:53.785221+0000","last_unstale":"2024-05-04T18:18:46.312727+0000","last_undegraded":"2024-05-04T18:18:46.312727+0000","last_fullsized":"2024-05-04T18:18:46.312727+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:28:11.181195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":2274,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.580658+0000","last_change":"2024-05-04T17:46:43.226742+0000","last_active":"2024-05-04T18:18:47.580658+0000","last_peered":"2024-05-04T18:18:47.580658+0000","last_clean":"2024-05-04T18:18:47.580658+0000","last_became_active":"2024-05-04T17:46:43.193128+0000","last_became_peered":"2024-05-04T17:46:43.193128+0000","last_unstale":"2024-05-04T18:18:47.580658+0000","last_undegraded":"2024-05-04T18:18:47.580658+0000","last_fullsized":"2024-05-04T18:18:47.580658+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:52:43.388505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":2283,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.239474+0000","last_change":"2024-05-04T17:46:42.337619+0000","last_active":"2024-05-04T18:17:25.239474+0000","last_peered":"2024-05-04T18:17:25.239474+0000","last_clean":"2024-05-04T18:17:25.239474+0000","last_became_active":"2024-05-04T17:46:42.332321+0000","last_became_peered":"2024-05-04T17:46:42.332321+0000","last_unstale":"2024-05-04T18:17:25.239474+0000","last_undegraded":"2024-05-04T18:17:25.239474+0000","last_fullsized":"2024-05-04T18:17:25.239474+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:45:24.298610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":2262,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.637251+0000","last_change":"2024-05-04T17:47:27.489125+0000","last_active":"2024-05-04T18:18:47.637251+0000","last_peered":"2024-05-04T18:18:47.637251+0000","last_clean":"2024-05-04T18:18:47.637251+0000","last_became_active":"2024-05-04T17:47:27.462872+0000","last_became_peered":"2024-05-04T17:47:27.462872+0000","last_unstale":"2024-05-04T18:18:47.637251+0000","last_undegraded":"2024-05-04T18:18:47.637251+0000","last_fullsized":"2024-05-04T18:18:47.637251+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:31:19.507264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":2241,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.620547+0000","last_change":"2024-05-04T17:47:04.137847+0000","last_active":"2024-05-04T18:18:47.620547+0000","last_peered":"2024-05-04T18:18:47.620547+0000","last_clean":"2024-05-04T18:18:47.620547+0000","last_became_active":"2024-05-04T17:47:04.099185+0000","last_became_peered":"2024-05-04T17:47:04.099185+0000","last_unstale":"2024-05-04T18:18:47.620547+0000","last_undegraded":"2024-05-04T18:18:47.620547+0000","last_fullsized":"2024-05-04T18:18:47.620547+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:52:57.980272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":2272,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-05-04T18:17:25.270653+0000","last_change":"2024-05-04T17:47:15.314211+0000","last_active":"2024-05-04T18:17:25.270653+0000","last_peered":"2024-05-04T18:17:25.270653+0000","last_clean":"2024-05-04T18:17:25.270653+0000","last_became_active":"2024-05-04T17:47:15.307302+0000","last_became_peered":"2024-05-04T17:47:15.307302+0000","last_unstale":"2024-05-04T18:17:25.270653+0000","last_undegraded":"2024-05-04T18:17:25.270653+0000","last_fullsized":"2024-05-04T18:17:25.270653+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:21:22.778712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":2267,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.627221+0000","last_change":"2024-05-04T17:47:40.632890+0000","last_active":"2024-05-04T18:18:47.627221+0000","last_peered":"2024-05-04T18:18:47.627221+0000","last_clean":"2024-05-04T18:18:47.627221+0000","last_became_active":"2024-05-04T17:47:40.627379+0000","last_became_peered":"2024-05-04T17:47:40.627379+0000","last_unstale":"2024-05-04T18:18:47.627221+0000","last_undegraded":"2024-05-04T18:18:47.627221+0000","last_fullsized":"2024-05-04T18:18:47.627221+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:24:44.506896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":2241,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.590521+0000","last_change":"2024-05-04T17:47:33.306252+0000","last_active":"2024-05-04T18:18:47.590521+0000","last_peered":"2024-05-04T18:18:47.590521+0000","last_clean":"2024-05-04T18:18:47.590521+0000","last_became_active":"2024-05-04T17:47:33.296115+0000","last_became_peered":"2024-05-04T17:47:33.296115+0000","last_unstale":"2024-05-04T18:18:47.590521+0000","last_undegraded":"2024-05-04T18:18:47.590521+0000","last_fullsized":"2024-05-04T18:18:47.590521+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:49:52.193030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":2275,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.546231+0000","last_change":"2024-05-04T17:46:42.745978+0000","last_active":"2024-05-04T18:18:47.546231+0000","last_peered":"2024-05-04T18:18:47.546231+0000","last_clean":"2024-05-04T18:18:47.546231+0000","last_became_active":"2024-05-04T17:46:42.724604+0000","last_became_peered":"2024-05-04T17:46:42.724604+0000","last_unstale":"2024-05-04T18:18:47.546231+0000","last_undegraded":"2024-05-04T18:18:47.546231+0000","last_fullsized":"2024-05-04T18:18:47.546231+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:52:59.865599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":2279,"reported_epoch":1184,"state":"active+clean","last_fresh":"2024-05-04T18:18:46.288621+0000","last_change":"2024-05-04T17:46:45.356387+0000","last_active":"2024-05-04T18:18:46.288621+0000","last_peered":"2024-05-04T18:18:46.288621+0000","last_clean":"2024-05-04T18:18:46.288621+0000","last_became_active":"2024-05-04T17:46:45.347368+0000","last_became_peered":"2024-05-04T17:46:45.347368+0000","last_unstale":"2024-05-04T18:18:46.288621+0000","last_undegraded":"2024-05-04T18:18:46.288621+0000","last_fullsized":"2024-05-04T18:18:46.288621+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:10:48.903095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":2268,"reported_epoch":1185,"state":"active+clean","last_fresh":"2024-05-04T18:18:47.570558+0000","last_change":"2024-05-04T17:47:38.797623+0000","last_active":"2024-05-04T18:18:47.570558+0000","last_peered":"2024-05-04T18:18:47.570558+0000","last_clean":"2024-05-04T18:18:47.570558+0000","last_became_active":"2024-05-04T17:47:38.791956+0000","last_became_peered":"2024-05-04T17:47:38.791956+0000","last_unstale":"2024-05-04T18:18:47.570558+0000","last_undegraded":"2024-05-04T18:18:47.570558+0000","last_fullsized":"2024-05-04T18:18:47.570558+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:47:30.587754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313621,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66692,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305148,"statfs":{"total":96636764160,"available":96568471552,"internally_reserved":0,"allocated":3166208,"data_stored":1353769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[]},{"osd":0,"up_from":20,"seq":85899346321,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66508,"kb_used_data":3152,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305332,"statfs":{"total":96636764160,"available":96568659968,"internally_reserved":0,"allocated":3227648,"data_stored":1224566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379029,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66272,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305568,"statfs":{"total":96636764160,"available":96568901632,"internally_reserved":0,"allocated":2985984,"data_stored":1082307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":4,"up_from":19,"seq":81604379030,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66268,"kb_used_data":2912,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305572,"statfs":{"total":96636764160,"available":96568905728,"internally_reserved":0,"allocated":2981888,"data_stored":1082302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":16,"seq":68719477133,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69980,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301860,"statfs":{"total":96636764160,"available":96565104640,"internally_reserved":0,"allocated":2523136,"data_stored":818417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542547,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66064,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305776,"statfs":{"total":96636764160,"available":96569114624,"internally_reserved":0,"allocated":2523136,"data_stored":814326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542543,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69976,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301864,"statfs":{"total":96636764160,"available":96565108736,"internally_reserved":0,"allocated":2519040,"data_stored":814321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575251,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70600,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301240,"statfs":{"total":96636764160,"available":96564469760,"internally_reserved":0,"allocated":2908160,"data_stored":1205411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T18:18:50.893 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-04T18:18:50.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.992+0000 7fe6a46c1640 10 monclient: get_monmap_and_config 2024-05-04T18:18:50.993 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.992+0000 7fe6a46c1640 10 monclient: build_initial_monmap 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.992+0000 7fe6a46c1640 10 monclient: monmap: 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:50.994163+0000 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:50.994163+0000 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:50.994 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:50.995 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.992+0000 7fe6a46c1640 1 Processor -- start 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 1 -- start start 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 10 monclient: init 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe69c0a9720 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fe69c0b2ec0 0x7fe69c13fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 10 monclient(hunting): picked mon.noname-a con 0x7fe69c0b2ec0 addr v2:172.21.15.19:3300/0 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe69c140270 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe69c0ab6f0 con 0x7fe69c140270 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe69c0b3320 con 0x7fe69c0a9720 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fe69c0a9b00 con 0x7fe69c0b2ec0 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.993+0000 7fe6a46c1640 10 monclient(hunting): authenticate will time out at 3353.108382s 2024-05-04T18:18:50.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.994+0000 7fe6a1c5c640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fe69c0b2ec0 0x7fe69c13fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:50.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.994+0000 7fe6a245d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe69c0a9720 0x7fe69c146a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:44464/0 (socket says 172.21.15.19:44464) 2024-05-04T18:18:50.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.994+0000 7fe6a245d640 1 -- 172.21.15.19:0/2432219417 learned_addr learned my addr 172.21.15.19:0/2432219417 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T18:18:50.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.994+0000 7fe6a1c5c640 10 monclient(hunting): get_auth_request con 0x7fe69c0b2ec0 auth_method 0 2024-05-04T18:18:50.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.994+0000 7fe6a1c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:50.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.994+0000 7fe6a1c5c640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:50.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.994+0000 7fe6a1c5c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:50.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.995+0000 7fe6a1c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:50.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.995+0000 7fe6a1c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:50.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.995+0000 7fe6a1c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:50.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.996+0000 7fe6a145b640 1 -- 172.21.15.19:0/2432219417 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 910477566 0 0) 0x7fe69c0b3320 con 0x7fe69c0a9720 2024-05-04T18:18:50.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.996+0000 7fe6a145b640 10 monclient(hunting): _negotiate 2024-05-04T18:18:50.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.996+0000 7fe6a145b640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:50.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.997+0000 7fe6a145b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:50.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.997+0000 7fe6a145b640 10 monclient(hunting): my global_id is 22726 2024-05-04T18:18:50.998 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.997+0000 7fe6a145b640 1 -- 172.21.15.19:0/2432219417 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe688003520 con 0x7fe69c0a9720 2024-05-04T18:18:50.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.997+0000 7fe6a1c5c640 10 monclient(hunting): handle_auth_done global_id 15750 payload 274 2024-05-04T18:18:50.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.997+0000 7fe6a1c5c640 1 -- 172.21.15.19:0/2432219417 >> v1:172.21.15.186:6789/0 conn(0x7fe69c0a9720 legacy=0x7fe69c146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:50.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.997+0000 7fe6a1c5c640 1 -- 172.21.15.19:0/2432219417 >> v1:172.21.15.19:6789/0 conn(0x7fe69c140270 legacy=0x7fe69c14d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:50.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.997+0000 7fe6a1c5c640 10 monclient: _finish_hunting 0 2024-05-04T18:18:50.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.997+0000 7fe6a1c5c640 1 monclient: found mon.noname-a 2024-05-04T18:18:50.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.997+0000 7fe6a1c5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:50.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.997+0000 7fe6a1c5c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T18:18:50.999 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:50.997+0000 7fe6a1c5c640 1 -- 172.21.15.19:0/2432219417 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69c0ab960 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.003 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.002+0000 7fe6a1c5c640 1 --2- 172.21.15.19:0/2432219417 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe69c0b2ec0 0x7fe69c13fd30 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0097a0 tx=0x7fe68c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc4a03d8cd60fca6 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:51.007 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 1 -- 172.21.15.19:0/2432219417 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c035020 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 10 monclient: dump: 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 10 monclient: _finish_auth 0 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 10 monclient: _check_auth_tickets 2024-05-04T18:18:51.008 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 1 -- 172.21.15.19:0/2432219417 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe68c031830 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 1 -- 172.21.15.19:0/2432219417 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c031c00 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a46c1640 5 monclient: authenticate success, global_id 15750 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a46c1640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a46c1640 4 set_mon_vals no callback set 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.006+0000 7fe6a145b640 10 monclient: dump: 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:51.009 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:51.010 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:51.010 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:51.010 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:51.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.007+0000 7fe6a46c1640 10 monclient: shutdown 2024-05-04T18:18:51.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.007+0000 7fe6a46c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:51.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.007+0000 7fe6a46c1640 1 -- 172.21.15.19:0/2432219417 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe69c0b2ec0 msgr2=0x7fe69c13fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:51.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.007+0000 7fe6a46c1640 1 --2- 172.21.15.19:0/2432219417 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe69c0b2ec0 0x7fe69c13fd30 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0097a0 tx=0x7fe68c02b8d0 comp rx=0 tx=0).stop 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.007+0000 7fe6a46c1640 1 -- 172.21.15.19:0/2432219417 shutdown_connections 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.007+0000 7fe6a46c1640 1 --2- 172.21.15.19:0/2432219417 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe69c0b2ec0 0x7fe69c13fd30 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.007+0000 7fe6a46c1640 1 -- 172.21.15.19:0/2432219417 >> 172.21.15.19:0/2432219417 conn(0x7fe69c0b84c0 msgr2=0x7fe69c0b88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.007+0000 7fe6a46c1640 1 -- 172.21.15.19:0/2432219417 shutdown_connections 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.007+0000 7fe6a46c1640 1 -- 172.21.15.19:0/2432219417 wait complete. 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.007+0000 7fe6a46c1640 10 monclient: build_initial_monmap 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.007+0000 7fe6a46c1640 10 monclient: monmap: 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:51.013 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.008+0000 7fe6a46c1640 1 librados: starting msgr at 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.008+0000 7fe6a46c1640 1 librados: starting objecter 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.008+0000 7fe6a46c1640 1 Processor -- start 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.008+0000 7fe6a46c1640 1 -- start start 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.008+0000 7fe6a46c1640 1 librados: setting wanted keys 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.008+0000 7fe6a46c1640 1 librados: calling monclient init 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.008+0000 7fe6a46c1640 10 monclient: init 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.008+0000 7fe6a46c1640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.008+0000 7fe6a46c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.008+0000 7fe6a46c1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe69c0a9720 0x7fe69c0b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.008+0000 7fe6a46c1640 10 monclient(hunting): picked mon.noname-b con 0x7fe69c0a9720 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.008+0000 7fe6a46c1640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe69c0b2ec0 0x7fe69c1db800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a46c1640 10 monclient(hunting): picked mon.noname-c con 0x7fe69c0b2ec0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:51.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a46c1640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe69c140270 0x7fe69c0b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a46c1640 10 monclient(hunting): picked mon.noname-a con 0x7fe69c140270 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a46c1640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a46c1640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c0b28b0 con 0x7fe69c140270 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a46c1640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a46c1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c0a9da0 con 0x7fe69c0a9720 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a46c1640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a46c1640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fe69c1406a0 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a46c1640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a46c1640 10 monclient(hunting): authenticate will time out at 3353.123676s 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a1c5c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe69c0b2ec0 0x7fe69c1db800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a2c5e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe69c140270 0x7fe69c0b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.009+0000 7fe6a1c5c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe69c0b2ec0 0x7fe69c1db800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58080/0 (socket says 172.21.15.19:58080) 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a1c5c640 1 -- 172.21.15.19:0/250365432 learned_addr learned my addr 172.21.15.19:0/250365432 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a1c5c640 10 monclient(hunting): get_auth_request con 0x7fe69c0b2ec0 auth_method 0 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a1c5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a1c5c640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:51.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a1c5c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:51.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a2c5e640 10 monclient(hunting): get_auth_request con 0x7fe69c140270 auth_method 0 2024-05-04T18:18:51.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a2c5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:51.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a2c5e640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:51.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a2c5e640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:51.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a1c5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:51.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a1c5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:51.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a1c5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:51.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a2c5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:51.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a2c5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:51.016 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.010+0000 7fe6a2c5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.013+0000 7fe6a1c5c640 10 monclient(hunting): handle_auth_done global_id 18773 payload 824 2024-05-04T18:18:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.013+0000 7fe6a1c5c640 1 -- 172.21.15.19:0/250365432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe69c0a9720 msgr2=0x7fe69c0b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.013+0000 7fe6a1c5c640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe69c0a9720 0x7fe69c0b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.013+0000 7fe6a1c5c640 1 -- 172.21.15.19:0/250365432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe69c140270 msgr2=0x7fe69c0b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.013+0000 7fe6a1c5c640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe69c140270 0x7fe69c0b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.013+0000 7fe6a1c5c640 10 monclient: _finish_hunting 0 2024-05-04T18:18:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.013+0000 7fe6a1c5c640 1 monclient: found mon.noname-c 2024-05-04T18:18:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.013+0000 7fe6a1c5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.013+0000 7fe6a1c5c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T18:18:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.013+0000 7fe6a1c5c640 1 -- 172.21.15.19:0/250365432 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69c1dbd40 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.013+0000 7fe6a2c5e640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe69c140270 0x7fe69c0b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-04T18:18:51.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.015+0000 7fe6a1c5c640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe69c0b2ec0 0x7fe69c1db800 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7fe68c02bee0 tx=0x7fe68c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:51.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.015+0000 7fe692ffd640 10 client.?.objecter ms_handle_connect 0x7fe69c0b2ec0 2024-05-04T18:18:51.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.015+0000 7fe692ffd640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:51.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 1 -- 172.21.15.19:0/250365432 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c041070 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:51.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T18:18:51.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 10 monclient: dump: 2024-05-04T18:18:51.021 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:51.021 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:51.021 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:51.021 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:51.021 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 10 monclient: _finish_auth 0 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 10 monclient: _check_auth_tickets 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 1 -- 172.21.15.19:0/250365432 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe68c03b920 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe6a46c1640 5 monclient: authenticate success, global_id 18773 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 1 -- 172.21.15.19:0/250365432 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c035040 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe6a46c1640 10 monclient: _renew_subs 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe6a46c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe6a46c1640 1 -- 172.21.15.19:0/250365432 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe69c143110 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.022 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe693fff640 4 set_mon_vals no callback set 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe692ffd640 10 monclient: dump: 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe6a46c1640 10 client.18773.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe6a46c1640 10 monclient: _renew_subs 2024-05-04T18:18:51.023 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe6a46c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:51.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe6a46c1640 1 -- 172.21.15.19:0/250365432 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe69c0b0c60 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.019+0000 7fe6a46c1640 1 librados: init done 2024-05-04T18:18:51.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.020+0000 7fe6a46c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:51.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.020+0000 7fe6a46c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:51.024 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.020+0000 7fe6a46c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:51.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.020+0000 7fe6a46c1640 1 -- 172.21.15.19:0/250365432 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe69c1dc050 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.299+0000 7fe692ffd640 1 -- 172.21.15.19:0/250365432 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe68c04c060 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.299+0000 7fe692ffd640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe68006d7c0 0x7fe68006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:51.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.299+0000 7fe6a245d640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe68006d7c0 0x7fe68006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:51.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.299+0000 7fe6a245d640 10 monclient: get_auth_request con 0x7fe68006d7c0 auth_method 0 2024-05-04T18:18:51.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.300+0000 7fe6a245d640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe68006d7c0 0x7fe68006fc80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe698000f60 tx=0x7fe6980078f0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:51.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.300+0000 7fe692ffd640 10 client.18773.objecter ms_handle_connect 0x7fe68006d7c0 2024-05-04T18:18:51.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.313+0000 7fe692ffd640 1 -- 172.21.15.19:0/250365432 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7fe68c078f70 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.313+0000 7fe692ffd640 10 client.18773.objecter ms_dispatch 0x7fe69c000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:51.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.313+0000 7fe692ffd640 3 client.18773.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:51.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.313+0000 7fe692ffd640 3 client.18773.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:51.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.313+0000 7fe692ffd640 20 client.18773.objecter dump_active .. 0 homeless 2024-05-04T18:18:51.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.313+0000 7fe692ffd640 1 -- 172.21.15.19:0/250365432 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe68c07b5d0 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.313+0000 7fe692ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:51.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.313+0000 7fe692ffd640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:18:51.522 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.521+0000 7fe6a46c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-05-04T18:18:51.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.521+0000 7fe6a46c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T18:18:51.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.521+0000 7fe6a46c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:51.523 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.521+0000 7fe6a46c1640 1 -- 172.21.15.19:0/250365432 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe69c143310 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.562+0000 7fe692ffd640 1 -- 172.21.15.19:0/250365432 <== mon.2 v2:172.21.15.19:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1186) ==== 74+0+17873 (secure 0 0 0) 0x7fe68c07aed0 con 0x7fe69c0b2ec0 2024-05-04T18:18:51.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.562+0000 7fe692ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-05-04T18:18:51.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.562+0000 7fe692ffd640 10 monclient: _finish_command 2 = system:0 2024-05-04T18:18:51.564 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T18:18:51.564 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":1186,"fsid":"18ced2df-a29f-46cc-b71a-1855bfb60c47","created":"2024-05-04T17:44:16.995866+0000","modified":"2024-05-04T18:18:48.342040+0000","last_up_change":"2024-05-04T17:45:15.621507+0000","last_in_change":"2024-05-04T17:44:23.464900+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":117,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-04T17:45:03.247977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-05-04T17:45:21.183165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-05-04T17:46:23.032749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4b749ec0-22b6-458c-a7af-2ccace806ab7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6808","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6809","nonce":2236557705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6810","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6811","nonce":2236557705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6814","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6815","nonce":2236557705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6812","nonce":2236557705},{"type":"v1","addr":"172.21.15.19:6813","nonce":2236557705}]},"public_addr":"172.21.15.19:6809/2236557705","cluster_addr":"172.21.15.19:6811/2236557705","heartbeat_back_addr":"172.21.15.19:6815/2236557705","heartbeat_front_addr":"172.21.15.19:6813/2236557705","state":["exists","up"]},{"osd":1,"uuid":"3aa520fe-39c4-4fa2-860b-f4d0cc04ca97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":1177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6800","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6801","nonce":3360656308}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6802","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6803","nonce":3360656308}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6806","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6807","nonce":3360656308}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6804","nonce":3360656308},{"type":"v1","addr":"172.21.15.19:6805","nonce":3360656308}]},"public_addr":"172.21.15.19:6801/3360656308","cluster_addr":"172.21.15.19:6803/3360656308","heartbeat_back_addr":"172.21.15.19:6807/3360656308","heartbeat_front_addr":"172.21.15.19:6805/3360656308","state":["exists","up"]},{"osd":2,"uuid":"16ed21c9-e5c3-4d65-b4d1-e4927a0fe919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6816","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6817","nonce":3744614279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6818","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6820","nonce":3744614279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6825","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6827","nonce":3744614279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6822","nonce":3744614279},{"type":"v1","addr":"172.21.15.19:6823","nonce":3744614279}]},"public_addr":"172.21.15.19:6817/3744614279","cluster_addr":"172.21.15.19:6820/3744614279","heartbeat_back_addr":"172.21.15.19:6827/3744614279","heartbeat_front_addr":"172.21.15.19:6823/3744614279","state":["exists","up"]},{"osd":3,"uuid":"45aa2c0f-cfae-478c-a60c-4c1491e45525","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6819","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6821","nonce":2537429684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6824","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6826","nonce":2537429684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6830","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6831","nonce":2537429684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.19:6828","nonce":2537429684},{"type":"v1","addr":"172.21.15.19:6829","nonce":2537429684}]},"public_addr":"172.21.15.19:6821/2537429684","cluster_addr":"172.21.15.19:6826/2537429684","heartbeat_back_addr":"172.21.15.19:6831/2537429684","heartbeat_front_addr":"172.21.15.19:6829/2537429684","state":["exists","up"]},{"osd":4,"uuid":"cb00f84f-d78c-461c-9843-73a9da4b80a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":1177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6819","nonce":356995042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6821","nonce":356995042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6825","nonce":356995042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":356995042},{"type":"v1","addr":"172.21.15.186:6823","nonce":356995042}]},"public_addr":"172.21.15.186:6819/356995042","cluster_addr":"172.21.15.186:6821/356995042","heartbeat_back_addr":"172.21.15.186:6825/356995042","heartbeat_front_addr":"172.21.15.186:6823/356995042","state":["exists","up"]},{"osd":5,"uuid":"d07ad403-291f-4bde-a64a-5181fb79e544","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6811","nonce":3728813703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6813","nonce":3728813703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6817","nonce":3728813703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":3728813703},{"type":"v1","addr":"172.21.15.186:6815","nonce":3728813703}]},"public_addr":"172.21.15.186:6811/3728813703","cluster_addr":"172.21.15.186:6813/3728813703","heartbeat_back_addr":"172.21.15.186:6817/3728813703","heartbeat_front_addr":"172.21.15.186:6815/3728813703","state":["exists","up"]},{"osd":6,"uuid":"5110d1ad-5634-4c8e-9338-d6c4229b1d63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":1177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6803","nonce":4252462313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6805","nonce":4252462313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6809","nonce":4252462313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":4252462313},{"type":"v1","addr":"172.21.15.186:6807","nonce":4252462313}]},"public_addr":"172.21.15.186:6803/4252462313","cluster_addr":"172.21.15.186:6805/4252462313","heartbeat_back_addr":"172.21.15.186:6809/4252462313","heartbeat_front_addr":"172.21.15.186:6807/4252462313","state":["exists","up"]},{"osd":7,"uuid":"b307b905-5cc8-447b-8765-d888ed6f655b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6827","nonce":1403900459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6829","nonce":1403900459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6833","nonce":1403900459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":1403900459},{"type":"v1","addr":"172.21.15.186:6831","nonce":1403900459}]},"public_addr":"172.21.15.186:6827/1403900459","cluster_addr":"172.21.15.186:6829/1403900459","heartbeat_back_addr":"172.21.15.186:6833/1403900459","heartbeat_front_addr":"172.21.15.186:6831/1403900459","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.360122+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.570945+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:01.075527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:02.428399+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.550872+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:44:59.984305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.443005+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-04T17:45:00.439608+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":5,"to":3}]},{"pgid":"14.e","mappings":[{"from":5,"to":0}]},{"pgid":"14.1f","mappings":[{"from":4,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":25,"snaps":[{"begin":1,"length":1}]},{"pool":28,"snaps":[{"begin":4,"length":3}]},{"pool":29,"snaps":[{"begin":4,"length":1}]},{"pool":68,"snaps":[{"begin":1,"length":1}]},{"pool":85,"snaps":[{"begin":1,"length":1}]},{"pool":90,"snaps":[{"begin":7,"length":2}]},{"pool":91,"snaps":[{"begin":1,"length":2}]},{"pool":98,"snaps":[{"begin":3,"length":1}]},{"pool":109,"snaps":[{"begin":2,"length":2}]},{"pool":125,"snaps":[{"begin":2,"length":3}]},{"pool":126,"snaps":[{"begin":3,"length":2}]},{"pool":136,"snaps":[{"begin":2,"length":2}]},{"pool":138,"snaps":[{"begin":2,"length":1}]},{"pool":139,"snaps":[{"begin":2,"length":4}]},{"pool":141,"snaps":[{"begin":3,"length":2}]},{"pool":171,"snaps":[{"begin":4,"length":4}]},{"pool":180,"snaps":[{"begin":4,"length":1}]},{"pool":243,"snaps":[{"begin":7,"length":1}]},{"pool":260,"snaps":[{"begin":8,"length":1}]},{"pool":265,"snaps":[{"begin":10,"length":1}]},{"pool":267,"snaps":[{"begin":11,"length":2}]},{"pool":270,"snaps":[{"begin":14,"length":1}]},{"pool":313,"snaps":[{"begin":2,"length":1}]},{"pool":314,"snaps":[{"begin":3,"length":1}]},{"pool":317,"snaps":[{"begin":4,"length":1}]},{"pool":321,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-04T18:18:51.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.564+0000 7fe6a46c1640 10 librados: watch_flush enter 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.564+0000 7fe6a46c1640 10 librados: watch_flush exit 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.564+0000 7fe6a46c1640 20 client.18773.objecter shutdown clearing up homeless session... 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.564+0000 7fe6a46c1640 10 client.18773.objecter successfully canceled tick 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.564+0000 7fe6a46c1640 1 -- 172.21.15.19:0/250365432 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe68006d7c0 msgr2=0x7fe68006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.564+0000 7fe6a46c1640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe68006d7c0 0x7fe68006fc80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe698000f60 tx=0x7fe6980078f0 comp rx=0 tx=0).stop 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.564+0000 7fe6a46c1640 10 monclient: shutdown 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.564+0000 7fe6a46c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.564+0000 7fe6a46c1640 1 -- 172.21.15.19:0/250365432 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe69c0b2ec0 msgr2=0x7fe69c1db800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.564+0000 7fe6a46c1640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe69c0b2ec0 0x7fe69c1db800 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7fe68c02bee0 tx=0x7fe68c0046c0 comp rx=0 tx=0).stop 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.564+0000 7fe6a46c1640 1 -- 172.21.15.19:0/250365432 shutdown_connections 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.565+0000 7fe6a46c1640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe68006d7c0 0x7fe68006fc80 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.565+0000 7fe6a46c1640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe69c140270 0x7fe69c0b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.565+0000 7fe6a46c1640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe69c0b2ec0 0x7fe69c1db800 unknown :-1 s=CLOSED pgs=3163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:51.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.565+0000 7fe6a46c1640 1 --2- 172.21.15.19:0/250365432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe69c0a9720 0x7fe69c0b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:51.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.565+0000 7fe6a46c1640 1 -- 172.21.15.19:0/250365432 >> 172.21.15.19:0/250365432 conn(0x7fe69c0b84c0 msgr2=0x7fe69c0b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:51.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.565+0000 7fe6a46c1640 1 -- 172.21.15.19:0/250365432 shutdown_connections 2024-05-04T18:18:51.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.565+0000 7fe6a46c1640 1 -- 172.21.15.19:0/250365432 wait complete. 2024-05-04T18:18:51.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:51.565+0000 7fe6a46c1640 1 librados: shutdown 2024-05-04T18:18:52.578 INFO:tasks.ceph:Scrubbing osd.0 2024-05-04T18:18:52.578 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-04T18:18:52.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.689+0000 7fe568da0640 10 monclient: get_monmap_and_config 2024-05-04T18:18:52.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.689+0000 7fe568da0640 10 monclient: build_initial_monmap 2024-05-04T18:18:52.690 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.689+0000 7fe568da0640 10 monclient: monmap: 2024-05-04T18:18:52.691 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:52.691 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:52.691 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:52.690914+0000 2024-05-04T18:18:52.691 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:52.690914+0000 2024-05-04T18:18:52.691 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:52.691 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:52.691 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:52.691 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:52.691 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:52.691 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:52.691 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:52.691 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:52.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.689+0000 7fe568da0640 1 Processor -- start 2024-05-04T18:18:52.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.689+0000 7fe568da0640 1 -- start start 2024-05-04T18:18:52.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.689+0000 7fe568da0640 10 monclient: init 2024-05-04T18:18:52.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:52.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fe5640b8e10 0x7fe5640aa470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 10 monclient(hunting): picked mon.noname-a con 0x7fe5640b8e10 addr v2:172.21.15.19:3300/0 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5640b2230 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe56413f8b0 0x7fe56413fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 10 monclient(hunting): picked mon.noname-b con 0x7fe56413f8b0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5640aa9b0 con 0x7fe5640b2230 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fe5640b91f0 con 0x7fe5640b8e10 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fe5640a9fe0 con 0x7fe56413f8b0 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.690+0000 7fe568da0640 10 monclient(hunting): authenticate will time out at 3354.805184s 2024-05-04T18:18:52.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.691+0000 7fe5637fe640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fe5640b8e10 0x7fe5640aa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.691+0000 7fe563fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe56413f8b0 0x7fe56413fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.691+0000 7fe563fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe56413f8b0 0x7fe56413fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58096/0 (socket says 172.21.15.19:58096) 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.691+0000 7fe563fff640 1 -- 172.21.15.19:0/3439829040 learned_addr learned my addr 172.21.15.19:0/3439829040 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.691+0000 7fe563fff640 10 monclient(hunting): get_auth_request con 0x7fe56413f8b0 auth_method 0 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.691+0000 7fe563fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.691+0000 7fe563fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.691+0000 7fe563fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.691+0000 7fe5637fe640 10 monclient(hunting): get_auth_request con 0x7fe5640b8e10 auth_method 0 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.691+0000 7fe5637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.691+0000 7fe5637fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.691+0000 7fe5637fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.692+0000 7fe5637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.692+0000 7fe5637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.692+0000 7fe5637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:52.695 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.692+0000 7fe563fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:52.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.692+0000 7fe563fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:52.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.692+0000 7fe563fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:52.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.693+0000 7fe5637fe640 10 monclient(hunting): handle_auth_done global_id 15759 payload 274 2024-05-04T18:18:52.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.693+0000 7fe5637fe640 1 -- 172.21.15.19:0/3439829040 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe56413f8b0 msgr2=0x7fe56413fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:52.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.693+0000 7fe5637fe640 1 --2- 172.21.15.19:0/3439829040 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe56413f8b0 0x7fe56413fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.693+0000 7fe5637fe640 1 -- 172.21.15.19:0/3439829040 >> v1:172.21.15.186:6789/0 conn(0x7fe5640b2230 legacy=0x7fe564145fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:52.696 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.693+0000 7fe5637fe640 10 monclient: _finish_hunting 0 2024-05-04T18:18:52.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.693+0000 7fe5637fe640 1 monclient: found mon.noname-a 2024-05-04T18:18:52.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.693+0000 7fe5637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:52.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.693+0000 7fe5637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T18:18:52.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.693+0000 7fe5637fe640 1 -- 172.21.15.19:0/3439829040 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5640a9cb0 con 0x7fe5640b8e10 2024-05-04T18:18:52.697 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.696+0000 7fe5637fe640 1 --2- 172.21.15.19:0/3439829040 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5640b8e10 0x7fe5640aa470 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7fe5540097a0 tx=0x7fe55402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7291bd98322753b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:52.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.699+0000 7fe5627fc640 1 -- 172.21.15.19:0/3439829040 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe554035020 con 0x7fe5640b8e10 2024-05-04T18:18:52.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.699+0000 7fe5627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:52.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.699+0000 7fe5627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T18:18:52.701 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.699+0000 7fe5627fc640 10 monclient: dump: 2024-05-04T18:18:52.701 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:52.701 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:52.701 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.701 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.701 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe5627fc640 10 monclient: _finish_auth 0 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe5627fc640 10 monclient: _check_auth_tickets 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe5627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe5627fc640 1 -- 172.21.15.19:0/3439829040 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe554031830 con 0x7fe5640b8e10 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 5 monclient: authenticate success, global_id 15759 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe5627fc640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 4 set_mon_vals no callback set 2024-05-04T18:18:52.702 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe5627fc640 1 -- 172.21.15.19:0/3439829040 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe554031c00 con 0x7fe5640b8e10 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe5627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe5627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe5627fc640 10 monclient: dump: 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:52.703 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:52.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 10 monclient: shutdown 2024-05-04T18:18:52.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:52.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 1 -- 172.21.15.19:0/3439829040 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5640b8e10 msgr2=0x7fe5640aa470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:52.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 1 --2- 172.21.15.19:0/3439829040 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5640b8e10 0x7fe5640aa470 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7fe5540097a0 tx=0x7fe55402b8d0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 1 -- 172.21.15.19:0/3439829040 shutdown_connections 2024-05-04T18:18:52.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 1 --2- 172.21.15.19:0/3439829040 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe56413f8b0 0x7fe56413fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 1 --2- 172.21.15.19:0/3439829040 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5640b8e10 0x7fe5640aa470 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 1 -- 172.21.15.19:0/3439829040 >> 172.21.15.19:0/3439829040 conn(0x7fe564067620 msgr2=0x7fe5640b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:52.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 1 -- 172.21.15.19:0/3439829040 shutdown_connections 2024-05-04T18:18:52.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.700+0000 7fe568da0640 1 -- 172.21.15.19:0/3439829040 wait complete. 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.701+0000 7fe568da0640 10 monclient: build_initial_monmap 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.701+0000 7fe568da0640 10 monclient: monmap: 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.701+0000 7fe568da0640 1 librados: starting msgr at 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.701+0000 7fe568da0640 1 librados: starting objecter 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.701+0000 7fe568da0640 1 Processor -- start 2024-05-04T18:18:52.708 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 1 -- start start 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 1 librados: setting wanted keys 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 1 librados: calling monclient init 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 10 monclient: init 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5640b2230 0x7fe5640ad390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 10 monclient(hunting): picked mon.noname-b con 0x7fe5640b2230 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5640b8e10 0x7fe5640acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 10 monclient(hunting): picked mon.noname-a con 0x7fe5640b8e10 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe56413f8b0 0x7fe564141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 10 monclient(hunting): picked mon.noname-c con 0x7fe56413f8b0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fe564140230 con 0x7fe5640b8e10 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:52.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe5640ab810 con 0x7fe5640b2230 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fe5641403b0 con 0x7fe56413f8b0 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.702+0000 7fe568da0640 10 monclient(hunting): authenticate will time out at 3354.817479s 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe563fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe56413f8b0 0x7fe564141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe562ffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5640b8e10 0x7fe5640acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe5637fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5640b2230 0x7fe5640ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe563fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe56413f8b0 0x7fe564141e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58100/0 (socket says 172.21.15.19:58100) 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe563fff640 1 -- 172.21.15.19:0/3288656258 learned_addr learned my addr 172.21.15.19:0/3288656258 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe563fff640 10 monclient(hunting): get_auth_request con 0x7fe56413f8b0 auth_method 0 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe563fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe563fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe563fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe562ffd640 10 monclient(hunting): get_auth_request con 0x7fe5640b8e10 auth_method 0 2024-05-04T18:18:52.710 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe562ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe562ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe5637fe640 10 monclient(hunting): get_auth_request con 0x7fe5640b2230 auth_method 0 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe5637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe5637fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.703+0000 7fe5637fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.704+0000 7fe5637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.704+0000 7fe5637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.704+0000 7fe5637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.704+0000 7fe563fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.704+0000 7fe563fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.704+0000 7fe563fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.704+0000 7fe562ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.704+0000 7fe562ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:52.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.704+0000 7fe562ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:52.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.707+0000 7fe5637fe640 10 monclient(hunting): handle_auth_done global_id 22732 payload 824 2024-05-04T18:18:52.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.707+0000 7fe5637fe640 1 -- 172.21.15.19:0/3288656258 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe56413f8b0 msgr2=0x7fe564141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:52.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.707+0000 7fe5637fe640 1 --2- 172.21.15.19:0/3288656258 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fe56413f8b0 0x7fe564141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.707+0000 7fe5637fe640 1 -- 172.21.15.19:0/3288656258 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5640b8e10 msgr2=0x7fe5640acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:52.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.707+0000 7fe5637fe640 1 --2- 172.21.15.19:0/3288656258 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fe5640b8e10 0x7fe5640acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.707+0000 7fe5637fe640 10 monclient: _finish_hunting 0 2024-05-04T18:18:52.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.707+0000 7fe5637fe640 1 monclient: found mon.noname-b 2024-05-04T18:18:52.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.707+0000 7fe5637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:52.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.707+0000 7fe5637fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:52.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.707+0000 7fe5637fe640 1 -- 172.21.15.19:0/3288656258 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5640adf80 con 0x7fe5640b2230 2024-05-04T18:18:52.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.709+0000 7fe5637fe640 1 --2- 172.21.15.19:0/3288656258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5640b2230 0x7fe5640ad390 secure :-1 s=READY pgs=5822 cs=0 l=1 rev1=1 crypto rx=0x7fe554033000 tx=0x7fe5540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:52.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.709+0000 7fe54bfff640 10 client.?.objecter ms_handle_connect 0x7fe5640b2230 2024-05-04T18:18:52.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.709+0000 7fe54bfff640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:52.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.712+0000 7fe54bfff640 1 -- 172.21.15.19:0/3288656258 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe554035040 con 0x7fe5640b2230 2024-05-04T18:18:52.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.712+0000 7fe54bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:52.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.712+0000 7fe54bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:52.715 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.712+0000 7fe54bfff640 10 monclient: dump: 2024-05-04T18:18:52.715 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:52.715 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:52.715 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.715 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.715 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:52.715 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:52.715 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:52.715 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.712+0000 7fe54bfff640 10 monclient: _finish_auth 0 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.712+0000 7fe54bfff640 10 monclient: _check_auth_tickets 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.712+0000 7fe54bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 5 monclient: authenticate success, global_id 22732 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe54bfff640 1 -- 172.21.15.19:0/3288656258 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe55403b920 con 0x7fe5640b2230 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe54bfff640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe54bfff640 1 -- 172.21.15.19:0/3288656258 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55403bd10 con 0x7fe5640b2230 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 10 monclient: _renew_subs 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 1 -- 172.21.15.19:0/3288656258 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5640abfd0 con 0x7fe5640b2230 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe54bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:52.716 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe54bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe54bfff640 10 monclient: dump: 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe561ffb640 4 set_mon_vals no callback set 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 10 client.22732.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 10 monclient: _renew_subs 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:52.717 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 1 -- 172.21.15.19:0/3288656258 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5641423b0 con 0x7fe5640b2230 2024-05-04T18:18:52.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 1 librados: init done 2024-05-04T18:18:52.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 10 client.22732.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:52.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 15 client.22732.objecter _session_command_op_assign -1 1 2024-05-04T18:18:52.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 20 client.22732.objecter _get_session osd=-1 returning homeless 2024-05-04T18:18:52.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 10 client.22732.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:52.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:52.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.713+0000 7fe568da0640 1 -- 172.21.15.19:0/3288656258 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe5640adc00 con 0x7fe5640b2230 2024-05-04T18:18:52.735 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.734+0000 7fe54bfff640 1 -- 172.21.15.19:0/3288656258 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe554038020 con 0x7fe5640b2230 2024-05-04T18:18:52.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.734+0000 7fe54bfff640 1 --2- 172.21.15.19:0/3288656258 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe55006d810 0x7fe55006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:52.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.734+0000 7fe562ffd640 1 --2- 172.21.15.19:0/3288656258 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe55006d810 0x7fe55006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:52.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.734+0000 7fe562ffd640 10 monclient: get_auth_request con 0x7fe55006d810 auth_method 0 2024-05-04T18:18:52.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.735+0000 7fe562ffd640 1 --2- 172.21.15.19:0/3288656258 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe55006d810 0x7fe55006fcd0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe5640b1f20 tx=0x7fe558009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:52.736 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.735+0000 7fe54bfff640 10 client.22732.objecter ms_handle_connect 0x7fe55006d810 2024-05-04T18:18:52.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 1 -- 172.21.15.19:0/3288656258 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7fe554079640 con 0x7fe5640b2230 2024-05-04T18:18:52.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 10 client.22732.objecter ms_dispatch 0x7fe564000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:52.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 3 client.22732.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:52.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 3 client.22732.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:52.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 10 client.22732.objecter checking command 1 2024-05-04T18:18:52.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 1 --2- 172.21.15.19:0/3288656258 >> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] conn(0x7fe550075670 0x7fe550077b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:52.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 20 client.22732.objecter _get_session s=0x7fe550074fa0 osd=0 3 2024-05-04T18:18:52.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 20 client.22732.objecter put_session s=0x7fe550074fa0 osd=0 3 2024-05-04T18:18:52.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 15 client.22732.objecter _session_command_op_remove -1 1 2024-05-04T18:18:52.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 20 client.22732.objecter _get_session s=0x7fe550074fa0 osd=0 3 2024-05-04T18:18:52.740 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 20 client.22732.objecter get_session s=0x7fe550074fa0 osd=0 3 2024-05-04T18:18:52.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 15 client.22732.objecter _session_command_op_assign 0 1 2024-05-04T18:18:52.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 20 client.22732.objecter put_session s=0x7fe550074fa0 osd=0 4 2024-05-04T18:18:52.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 10 client.22732.objecter _send_command 1 2024-05-04T18:18:52.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 1 -- 172.21.15.19:0/3288656258 --> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe5500781e0 con 0x7fe550075670 2024-05-04T18:18:52.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 20 client.22732.objecter dump_active .. 0 homeless 2024-05-04T18:18:52.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 1 -- 172.21.15.19:0/3288656258 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=1186) ==== 24+0+0 (secure 0 0 0) 0x7fe554077840 con 0x7fe5640b2230 2024-05-04T18:18:52.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.738+0000 7fe54bfff640 10 monclient: handle_get_version_reply finishing 1 version 1186 2024-05-04T18:18:52.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.739+0000 7fe563fff640 1 --2- 172.21.15.19:0/3288656258 >> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] conn(0x7fe550075670 0x7fe550077b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:52.741 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.739+0000 7fe563fff640 10 monclient: get_auth_request con 0x7fe550075670 auth_method 0 2024-05-04T18:18:52.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.741+0000 7fe563fff640 1 --2- 172.21.15.19:0/3288656258 >> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] conn(0x7fe550075670 0x7fe550077b10 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:52.743 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.742+0000 7fe54bfff640 10 client.22732.objecter ms_handle_connect 0x7fe550075670 2024-05-04T18:18:52.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.766+0000 7fe54bfff640 1 -- 172.21.15.19:0/3288656258 <== osd.0 v2:172.21.15.19:6808/2236557705 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe5500781e0 con 0x7fe550075670 2024-05-04T18:18:52.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.766+0000 7fe54bfff640 10 client.22732.objecter ms_dispatch 0x7fe564000b90 command_reply(tid 1: 0 ) 2024-05-04T18:18:52.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.766+0000 7fe54bfff640 10 client.22732.objecter _finish_command 1 = system:0 2024-05-04T18:18:52.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.766+0000 7fe54bfff640 20 client.22732.objecter put_session s=0x7fe550074fa0 osd=0 4 2024-05-04T18:18:52.768 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.766+0000 7fe54bfff640 15 client.22732.objecter _session_command_op_remove 0 1 2024-05-04T18:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.780+0000 7fe568da0640 10 client.22732.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-04T18:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.780+0000 7fe568da0640 15 client.22732.objecter _session_command_op_assign -1 2 2024-05-04T18:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.780+0000 7fe568da0640 20 client.22732.objecter _get_session s=0x7fe550074fa0 osd=0 3 2024-05-04T18:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.780+0000 7fe568da0640 20 client.22732.objecter put_session s=0x7fe550074fa0 osd=0 3 2024-05-04T18:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.780+0000 7fe568da0640 20 client.22732.objecter _get_session s=0x7fe550074fa0 osd=0 3 2024-05-04T18:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.780+0000 7fe568da0640 15 client.22732.objecter _session_command_op_remove -1 2 2024-05-04T18:18:52.781 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.780+0000 7fe568da0640 20 client.22732.objecter get_session s=0x7fe550074fa0 osd=0 3 2024-05-04T18:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.780+0000 7fe568da0640 15 client.22732.objecter _session_command_op_assign 0 2 2024-05-04T18:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.780+0000 7fe568da0640 20 client.22732.objecter put_session s=0x7fe550074fa0 osd=0 4 2024-05-04T18:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.780+0000 7fe568da0640 10 client.22732.objecter _send_command 2 2024-05-04T18:18:52.782 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.780+0000 7fe568da0640 1 -- 172.21.15.19:0/3288656258 --> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe56413eac0 con 0x7fe550075670 2024-05-04T18:18:52.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.795+0000 7fe54bfff640 1 -- 172.21.15.19:0/3288656258 <== osd.0 v2:172.21.15.19:6808/2236557705 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe56413eac0 con 0x7fe550075670 2024-05-04T18:18:52.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.795+0000 7fe54bfff640 10 client.22732.objecter ms_dispatch 0x7fe564000b90 command_reply(tid 2: 0 ) 2024-05-04T18:18:52.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.795+0000 7fe54bfff640 10 client.22732.objecter _finish_command 2 = system:0 2024-05-04T18:18:52.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.795+0000 7fe54bfff640 20 client.22732.objecter put_session s=0x7fe550074fa0 osd=0 4 2024-05-04T18:18:52.797 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.795+0000 7fe54bfff640 15 client.22732.objecter _session_command_op_remove 0 2 2024-05-04T18:18:52.797 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-05-04T18:18:52.797 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T18:18:52.797 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-05-04T18:18:52.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 10 librados: watch_flush enter 2024-05-04T18:18:52.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 10 librados: watch_flush exit 2024-05-04T18:18:52.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 10 client.22732.objecter close_session for osd.0 2024-05-04T18:18:52.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 1 -- 172.21.15.19:0/3288656258 >> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] conn(0x7fe550075670 msgr2=0x7fe550077b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:52.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 1 --2- 172.21.15.19:0/3288656258 >> [v2:172.21.15.19:6808/2236557705,v1:172.21.15.19:6809/2236557705] conn(0x7fe550075670 0x7fe550077b10 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 20 client.22732.objecter put_session s=0x7fe550074fa0 osd=0 1 2024-05-04T18:18:52.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 20 client.22732.objecter shutdown clearing up homeless session... 2024-05-04T18:18:52.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 10 client.22732.objecter successfully canceled tick 2024-05-04T18:18:52.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 1 -- 172.21.15.19:0/3288656258 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe55006d810 msgr2=0x7fe55006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:52.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 1 --2- 172.21.15.19:0/3288656258 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fe55006d810 0x7fe55006fcd0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe5640b1f20 tx=0x7fe558009040 comp rx=0 tx=0).stop 2024-05-04T18:18:52.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 10 monclient: shutdown 2024-05-04T18:18:52.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:52.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 1 -- 172.21.15.19:0/3288656258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5640b2230 msgr2=0x7fe5640ad390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:52.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 1 --2- 172.21.15.19:0/3288656258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5640b2230 0x7fe5640ad390 secure :-1 s=READY pgs=5822 cs=0 l=1 rev1=1 crypto rx=0x7fe554033000 tx=0x7fe5540046c0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe563fff640 1 -- 172.21.15.19:0/3288656258 reap_dead start 2024-05-04T18:18:52.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 1 -- 172.21.15.19:0/3288656258 shutdown_connections 2024-05-04T18:18:52.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 1 -- 172.21.15.19:0/3288656258 >> 172.21.15.19:0/3288656258 conn(0x7fe564067620 msgr2=0x7fe56413de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:52.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 1 -- 172.21.15.19:0/3288656258 shutdown_connections 2024-05-04T18:18:52.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 1 -- 172.21.15.19:0/3288656258 wait complete. 2024-05-04T18:18:52.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.796+0000 7fe568da0640 1 librados: shutdown 2024-05-04T18:18:52.808 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-04T18:18:52.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.910+0000 7f5cd7532640 10 monclient: get_monmap_and_config 2024-05-04T18:18:52.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.910+0000 7f5cd7532640 10 monclient: build_initial_monmap 2024-05-04T18:18:52.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.910+0000 7f5cd7532640 10 monclient: monmap: 2024-05-04T18:18:52.912 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:52.912 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:52.912 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:52.912776+0000 2024-05-04T18:18:52.912 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:52.912776+0000 2024-05-04T18:18:52.912 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:52.913 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:52.913 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:52.913 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:52.913 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:52.913 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:52.913 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:52.913 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:52.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.911+0000 7f5cd7532640 1 Processor -- start 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.911+0000 7f5cd7532640 1 -- start start 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.911+0000 7f5cd7532640 10 monclient: init 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5cd006dc90 0x7f5cd013fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 10 monclient(hunting): picked mon.noname-c con 0x7f5cd006dc90 addr v2:172.21.15.186:3300/0 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 0x7f5cd0140070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 10 monclient(hunting): picked mon.noname-b con 0x7f5cd006e070 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5cd01460a0 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cd006d060 con 0x7f5cd01460a0 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f5cd0071d90 con 0x7f5cd006dc90 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f5cd0140660 con 0x7f5cd006e070 2024-05-04T18:18:52.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:52.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd7532640 10 monclient(hunting): authenticate will time out at 3355.027080s 2024-05-04T18:18:52.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.912+0000 7f5cd4acd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 0x7f5cd0140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:52.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.913+0000 7f5cd52ce640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5cd006dc90 0x7f5cd013fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:52.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.913+0000 7f5cd4acd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 0x7f5cd0140070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58114/0 (socket says 172.21.15.19:58114) 2024-05-04T18:18:52.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.913+0000 7f5cd4acd640 1 -- 172.21.15.19:0/509626729 learned_addr learned my addr 172.21.15.19:0/509626729 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:52.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.913+0000 7f5cd4acd640 10 monclient(hunting): get_auth_request con 0x7f5cd006e070 auth_method 0 2024-05-04T18:18:52.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.913+0000 7f5cd4acd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:52.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.913+0000 7f5cd4acd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:52.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.913+0000 7f5cd4acd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:52.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.914+0000 7f5cd52ce640 10 monclient(hunting): get_auth_request con 0x7f5cd006dc90 auth_method 0 2024-05-04T18:18:52.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.914+0000 7f5cd52ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:52.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.914+0000 7f5cd52ce640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:52.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.914+0000 7f5cd52ce640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:52.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.914+0000 7f5cd4acd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:52.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.914+0000 7f5cd4acd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:52.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.914+0000 7f5cd4acd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:52.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.916+0000 7f5cd4acd640 10 monclient(hunting): handle_auth_done global_id 18782 payload 274 2024-05-04T18:18:52.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.916+0000 7f5cd4acd640 1 -- 172.21.15.19:0/509626729 >> v2:172.21.15.186:3300/0 conn(0x7f5cd006dc90 msgr2=0x7f5cd013fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:52.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.916+0000 7f5cd4acd640 1 --2- 172.21.15.19:0/509626729 >> v2:172.21.15.186:3300/0 conn(0x7f5cd006dc90 0x7f5cd013fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.916+0000 7f5cd4acd640 1 -- 172.21.15.19:0/509626729 >> v1:172.21.15.186:6789/0 conn(0x7f5cd01460a0 legacy=0x7f5cd0146550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:52.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.916+0000 7f5cd4acd640 10 monclient: _finish_hunting 0 2024-05-04T18:18:52.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.916+0000 7f5cd4acd640 1 monclient: found mon.noname-b 2024-05-04T18:18:52.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.916+0000 7f5cd4acd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:52.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.916+0000 7f5cd4acd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T18:18:52.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.916+0000 7f5cd4acd640 1 -- 172.21.15.19:0/509626729 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd0073790 con 0x7f5cd006e070 2024-05-04T18:18:52.918 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.916+0000 7f5cd52ce640 1 --2- 172.21.15.19:0/509626729 >> v2:172.21.15.186:3300/0 conn(0x7f5cd006dc90 0x7f5cd013fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-04T18:18:52.919 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.918+0000 7f5cd4acd640 1 --2- 172.21.15.19:0/509626729 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 0x7f5cd0140070 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009870 tx=0x7f5cc402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f42f68bad84f821 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:52.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 1 -- 172.21.15.19:0/509626729 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40317c0 con 0x7f5cd006e070 2024-05-04T18:18:52.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:52.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:52.924 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 10 monclient: dump: 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 10 monclient: _finish_auth 0 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 10 monclient: _check_auth_tickets 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5cd7532640 5 monclient: authenticate success, global_id 18782 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5cd7532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T18:18:52.925 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 1 -- 172.21.15.19:0/509626729 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cc4037070 con 0x7f5cd006e070 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5cd7532640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5cd7532640 4 set_mon_vals no callback set 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 1 -- 172.21.15.19:0/509626729 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4031b80 con 0x7f5cd006e070 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.922+0000 7f5ccffff640 10 monclient: dump: 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:52.926 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:52.927 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:52.927 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.923+0000 7f5cd7532640 10 monclient: shutdown 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.923+0000 7f5cd7532640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.923+0000 7f5cd7532640 1 -- 172.21.15.19:0/509626729 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 msgr2=0x7f5cd0140070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.923+0000 7f5cd7532640 1 --2- 172.21.15.19:0/509626729 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 0x7f5cd0140070 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009870 tx=0x7f5cc402b860 comp rx=0 tx=0).stop 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.923+0000 7f5cd7532640 1 -- 172.21.15.19:0/509626729 shutdown_connections 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.923+0000 7f5cd7532640 1 --2- 172.21.15.19:0/509626729 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 0x7f5cd0140070 unknown :-1 s=CLOSED pgs=3164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.923+0000 7f5cd7532640 1 --2- 172.21.15.19:0/509626729 >> v2:172.21.15.186:3300/0 conn(0x7f5cd006dc90 0x7f5cd013fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.924+0000 7f5cd7532640 1 -- 172.21.15.19:0/509626729 >> 172.21.15.19:0/509626729 conn(0x7f5cd00b6980 msgr2=0x7f5cd00b6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.924+0000 7f5cd7532640 1 -- 172.21.15.19:0/509626729 shutdown_connections 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.924+0000 7f5cd7532640 1 -- 172.21.15.19:0/509626729 wait complete. 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.924+0000 7f5cd7532640 10 monclient: build_initial_monmap 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.925+0000 7f5cd7532640 10 monclient: monmap: 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:52.929 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.925+0000 7f5cd7532640 1 librados: starting msgr at 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.925+0000 7f5cd7532640 1 librados: starting objecter 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.925+0000 7f5cd7532640 1 Processor -- start 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.925+0000 7f5cd7532640 1 -- start start 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.925+0000 7f5cd7532640 1 librados: setting wanted keys 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.925+0000 7f5cd7532640 1 librados: calling monclient init 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.925+0000 7f5cd7532640 10 monclient: init 2024-05-04T18:18:52.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.925+0000 7f5cd7532640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.925+0000 7f5cd7532640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.925+0000 7f5cd7532640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cd006dc90 0x7f5cd00b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 10 monclient(hunting): picked mon.noname-b con 0x7f5cd006dc90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 0x7f5cd0140920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 10 monclient(hunting): picked mon.noname-c con 0x7f5cd006e070 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5cd01460a0 0x7f5cd0140e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 10 monclient(hunting): picked mon.noname-a con 0x7f5cd01460a0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd014d5b0 con 0x7f5cd01460a0 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd014d430 con 0x7f5cd006dc90 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f5cd014d730 con 0x7f5cd006e070 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:52.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.926+0000 7f5cd7532640 10 monclient(hunting): authenticate will time out at 3355.040712s 2024-05-04T18:18:52.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.929+0000 7f5cd4acd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 0x7f5cd0140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:52.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.929+0000 7f5cd4acd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 0x7f5cd0140920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58118/0 (socket says 172.21.15.19:58118) 2024-05-04T18:18:52.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.929+0000 7f5cd4acd640 1 -- 172.21.15.19:0/2121634361 learned_addr learned my addr 172.21.15.19:0/2121634361 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:52.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.929+0000 7f5cd4acd640 10 monclient(hunting): get_auth_request con 0x7f5cd006e070 auth_method 0 2024-05-04T18:18:52.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.929+0000 7f5cd4acd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:52.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.929+0000 7f5cd4acd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:52.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.929+0000 7f5cd4acd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:52.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.965+0000 7f5cd5acf640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5cd01460a0 0x7f5cd0140e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:52.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.965+0000 7f5cd5acf640 10 monclient(hunting): get_auth_request con 0x7f5cd01460a0 auth_method 0 2024-05-04T18:18:52.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.965+0000 7f5cd5acf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:52.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.965+0000 7f5cd5acf640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:52.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.965+0000 7f5cd5acf640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:52.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.966+0000 7f5cd5acf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:52.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.966+0000 7f5cd5acf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:52.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.966+0000 7f5cd5acf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:52.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.969+0000 7f5cd5acf640 10 monclient(hunting): handle_auth_done global_id 15765 payload 824 2024-05-04T18:18:52.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.969+0000 7f5cd5acf640 1 -- 172.21.15.19:0/2121634361 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 msgr2=0x7f5cd0140920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:52.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.969+0000 7f5cd5acf640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 0x7f5cd0140920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.969+0000 7f5cd5acf640 1 -- 172.21.15.19:0/2121634361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cd006dc90 msgr2=0x7f5cd00b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:52.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.969+0000 7f5cd5acf640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cd006dc90 0x7f5cd00b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:52.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.969+0000 7f5cd5acf640 10 monclient: _finish_hunting 0 2024-05-04T18:18:52.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.969+0000 7f5cd5acf640 1 monclient: found mon.noname-a 2024-05-04T18:18:52.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.969+0000 7f5cd5acf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:52.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.969+0000 7f5cd5acf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T18:18:52.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.969+0000 7f5cd5acf640 1 -- 172.21.15.19:0/2121634361 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd0137360 con 0x7f5cd01460a0 2024-05-04T18:18:52.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.971+0000 7f5cd5acf640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5cd01460a0 0x7f5cd0140e60 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000c410 tx=0x7f5cc000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:52.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.971+0000 7f5ccdffb640 10 client.?.objecter ms_handle_connect 0x7f5cd01460a0 2024-05-04T18:18:52.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.971+0000 7f5ccdffb640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:52.978 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.976+0000 7f5ccdffb640 1 -- 172.21.15.19:0/2121634361 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0017070 con 0x7f5cd01460a0 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.976+0000 7f5ccdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.976+0000 7f5ccdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.976+0000 7f5ccdffb640 10 monclient: dump: 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:52.979 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.976+0000 7f5ccdffb640 10 monclient: _finish_auth 0 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.976+0000 7f5ccdffb640 10 monclient: _check_auth_tickets 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.976+0000 7f5ccdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5cd7532640 5 monclient: authenticate success, global_id 15765 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5ccdffb640 1 -- 172.21.15.19:0/2121634361 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cc00035f0 con 0x7f5cd01460a0 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5ccdffb640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5ccdffb640 1 -- 172.21.15.19:0/2121634361 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc00039e0 con 0x7f5cd01460a0 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5cd7532640 10 monclient: _renew_subs 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5cd7532640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5cd7532640 1 -- 172.21.15.19:0/2121634361 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cd01375d0 con 0x7f5cd01460a0 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5ccdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5ccdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5ccf7fe640 4 set_mon_vals no callback set 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5ccdffb640 10 monclient: dump: 2024-05-04T18:18:52.980 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5cd7532640 10 client.15765.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5cd7532640 10 monclient: _renew_subs 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5cd7532640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5cd7532640 1 -- 172.21.15.19:0/2121634361 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cd013cd50 con 0x7f5cd01460a0 2024-05-04T18:18:52.981 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.977+0000 7f5cd7532640 1 librados: init done 2024-05-04T18:18:52.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.978+0000 7f5cd7532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:52.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.978+0000 7f5cd7532640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:52.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.978+0000 7f5cd7532640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T18:18:52.982 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:52.978+0000 7f5cd7532640 1 -- 172.21.15.19:0/2121634361 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cd0068b10 con 0x7f5cd01460a0 2024-05-04T18:18:53.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.255+0000 7f5ccdffb640 1 -- 172.21.15.19:0/2121634361 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5cc001a070 con 0x7f5cd01460a0 2024-05-04T18:18:53.257 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.256+0000 7f5ccdffb640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5cbc06d810 0x7f5cbc06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:53.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.256+0000 7f5cd52ce640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5cbc06d810 0x7f5cbc06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:53.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.256+0000 7f5cd52ce640 10 monclient: get_auth_request con 0x7f5cbc06d810 auth_method 0 2024-05-04T18:18:53.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.256+0000 7f5ccdffb640 10 client.15765.objecter ms_handle_connect 0x7f5cbc06d810 2024-05-04T18:18:53.258 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.256+0000 7f5cd52ce640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5cbc06d810 0x7f5cbc06fcd0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8009eb0 tx=0x7f5cc80094a0 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:53.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.285+0000 7f5ccdffb640 1 -- 172.21.15.19:0/2121634361 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f5cc0056950 con 0x7f5cd01460a0 2024-05-04T18:18:53.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.285+0000 7f5ccdffb640 10 client.15765.objecter ms_dispatch 0x7f5cd0000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:53.286 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.285+0000 7f5ccdffb640 3 client.15765.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:53.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.285+0000 7f5ccdffb640 3 client.15765.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:53.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.285+0000 7f5ccdffb640 20 client.15765.objecter dump_active .. 0 homeless 2024-05-04T18:18:53.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.285+0000 7f5ccdffb640 1 -- 172.21.15.19:0/2121634361 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5cc0021020 con 0x7f5cd01460a0 2024-05-04T18:18:53.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.285+0000 7f5ccdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:53.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.285+0000 7f5ccdffb640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:18:53.477 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.476+0000 7f5cd7532640 1 -- 172.21.15.19:0/2121634361 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f5cd014d210 con 0x7f5cbc06d810 2024-05-04T18:18:53.478 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.476+0000 7f5ccdffb640 1 -- 172.21.15.19:0/2121634361 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5cd014d210 con 0x7f5cbc06d810 2024-05-04T18:18:53.478 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 0 to deep-scrub 2024-05-04T18:18:53.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.478+0000 7f5cd7532640 10 librados: watch_flush enter 2024-05-04T18:18:53.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.478+0000 7f5cd7532640 10 librados: watch_flush exit 2024-05-04T18:18:53.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.478+0000 7f5cd7532640 20 client.15765.objecter shutdown clearing up homeless session... 2024-05-04T18:18:53.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.478+0000 7f5cd7532640 10 client.15765.objecter successfully canceled tick 2024-05-04T18:18:53.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 -- 172.21.15.19:0/2121634361 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5cbc06d810 msgr2=0x7f5cbc06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5cbc06d810 0x7f5cbc06fcd0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8009eb0 tx=0x7f5cc80094a0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 10 monclient: shutdown 2024-05-04T18:18:53.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:53.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 -- 172.21.15.19:0/2121634361 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5cd01460a0 msgr2=0x7f5cd0140e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5cd01460a0 0x7f5cd0140e60 secure :-1 s=READY pgs=3063 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000c410 tx=0x7f5cc000c8d0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 -- 172.21.15.19:0/2121634361 shutdown_connections 2024-05-04T18:18:53.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f5cbc06d810 0x7f5cbc06fcd0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f5cd01460a0 0x7f5cd0140e60 unknown :-1 s=CLOSED pgs=3063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f5cd006e070 0x7f5cd0140920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 --2- 172.21.15.19:0/2121634361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5cd006dc90 0x7f5cd00b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 -- 172.21.15.19:0/2121634361 >> 172.21.15.19:0/2121634361 conn(0x7f5cd00b6980 msgr2=0x7f5cd0072100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:53.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 -- 172.21.15.19:0/2121634361 shutdown_connections 2024-05-04T18:18:53.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 -- 172.21.15.19:0/2121634361 wait complete. 2024-05-04T18:18:53.482 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.479+0000 7f5cd7532640 1 librados: shutdown 2024-05-04T18:18:53.492 INFO:tasks.ceph:Scrubbing osd.1 2024-05-04T18:18:53.492 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-04T18:18:53.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.595+0000 7f3208d15640 10 monclient: get_monmap_and_config 2024-05-04T18:18:53.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.595+0000 7f3208d15640 10 monclient: build_initial_monmap 2024-05-04T18:18:53.597 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.595+0000 7f3208d15640 10 monclient: monmap: 2024-05-04T18:18:53.597 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:53.597 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:53.597 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:53.597442+0000 2024-05-04T18:18:53.597 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:53.597442+0000 2024-05-04T18:18:53.597 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:53.597 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:53.597 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:53.597 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:53.597 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:53.598 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:53.598 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:53.598 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:53.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.596+0000 7f3208d15640 1 Processor -- start 2024-05-04T18:18:53.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.596+0000 7f3208d15640 1 -- start start 2024-05-04T18:18:53.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.596+0000 7f3208d15640 10 monclient: init 2024-05-04T18:18:53.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.596+0000 7f3208d15640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:53.599 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.596+0000 7f3208d15640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.596+0000 7f3208d15640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f32040a9fb0 0x7f32040aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.596+0000 7f3208d15640 10 monclient(hunting): picked mon.noname-a con 0x7f32040a9fb0 addr v2:172.21.15.19:3300/0 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.597+0000 7f3208d15640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32040b27f0 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.597+0000 7f3208d15640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32041404e0 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.597+0000 7f3208d15640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32040ab880 con 0x7f32041404e0 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.597+0000 7f3208d15640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3204146b00 con 0x7f32040b27f0 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.597+0000 7f3208d15640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.597+0000 7f3208d15640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f3204146cd0 con 0x7f32040a9fb0 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.597+0000 7f3208d15640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.597+0000 7f3208d15640 10 monclient(hunting): authenticate will time out at 3355.711675s 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.597+0000 7f320259c640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f32040a9fb0 0x7f32040aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.597+0000 7f3201d9b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f32040b27f0 0x7f320413fdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:44492/0 (socket says 172.21.15.19:44492) 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.597+0000 7f3201d9b640 1 -- 172.21.15.19:0/1692387152 learned_addr learned my addr 172.21.15.19:0/1692387152 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.598+0000 7f320259c640 10 monclient(hunting): get_auth_request con 0x7f32040a9fb0 auth_method 0 2024-05-04T18:18:53.600 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.598+0000 7f320259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:53.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.598+0000 7f320259c640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:53.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.598+0000 7f320259c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:53.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.599+0000 7f320259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:53.601 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.599+0000 7f320259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:53.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.599+0000 7f320259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:53.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.599+0000 7f320159a640 1 -- 172.21.15.19:0/1692387152 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1092956854 0 0) 0x7f3204146b00 con 0x7f32040b27f0 2024-05-04T18:18:53.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.599+0000 7f320159a640 10 monclient(hunting): _negotiate 2024-05-04T18:18:53.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.599+0000 7f320159a640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:53.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.599+0000 7f320159a640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:53.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.599+0000 7f320159a640 10 monclient(hunting): my global_id is 22738 2024-05-04T18:18:53.602 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.599+0000 7f320159a640 1 -- 172.21.15.19:0/1692387152 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31ec003520 con 0x7f32040b27f0 2024-05-04T18:18:53.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.601+0000 7f320259c640 10 monclient(hunting): handle_auth_done global_id 15768 payload 274 2024-05-04T18:18:53.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.601+0000 7f320259c640 1 -- 172.21.15.19:0/1692387152 >> v1:172.21.15.186:6789/0 conn(0x7f32040b27f0 legacy=0x7f320413fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.601+0000 7f320259c640 1 -- 172.21.15.19:0/1692387152 >> v1:172.21.15.19:6789/0 conn(0x7f32041404e0 legacy=0x7f32041463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.601+0000 7f320259c640 10 monclient: _finish_hunting 0 2024-05-04T18:18:53.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.601+0000 7f320259c640 1 monclient: found mon.noname-a 2024-05-04T18:18:53.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.601+0000 7f320259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:53.603 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.601+0000 7f320259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T18:18:53.604 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.601+0000 7f320259c640 1 -- 172.21.15.19:0/1692387152 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3204146e50 con 0x7f32040a9fb0 2024-05-04T18:18:53.606 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.605+0000 7f320259c640 1 --2- 172.21.15.19:0/1692387152 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32040a9fb0 0x7f32040aa390 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f31f40097a0 tx=0x7f31f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8689dbb770e383 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:53.611 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 1 -- 172.21.15.19:0/1692387152 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f4035020 con 0x7f32040a9fb0 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 10 monclient: dump: 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:53.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 10 monclient: _finish_auth 0 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 10 monclient: _check_auth_tickets 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 1 -- 172.21.15.19:0/1692387152 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31f4031830 con 0x7f32040a9fb0 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 1 -- 172.21.15.19:0/1692387152 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f4031c00 con 0x7f32040a9fb0 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f320159a640 10 monclient: dump: 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:53.613 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:53.614 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:53.614 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:53.614 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:53.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f3208d15640 5 monclient: authenticate success, global_id 15768 2024-05-04T18:18:53.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f3208d15640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:53.614 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.610+0000 7f3208d15640 4 set_mon_vals no callback set 2024-05-04T18:18:53.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.611+0000 7f3208d15640 10 monclient: shutdown 2024-05-04T18:18:53.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.611+0000 7f3208d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:53.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.611+0000 7f3208d15640 1 -- 172.21.15.19:0/1692387152 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32040a9fb0 msgr2=0x7f32040aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.611+0000 7f3208d15640 1 --2- 172.21.15.19:0/1692387152 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32040a9fb0 0x7f32040aa390 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f31f40097a0 tx=0x7f31f402b8d0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.611+0000 7f3208d15640 1 -- 172.21.15.19:0/1692387152 shutdown_connections 2024-05-04T18:18:53.617 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.611+0000 7f3208d15640 1 --2- 172.21.15.19:0/1692387152 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32040a9fb0 0x7f32040aa390 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.611+0000 7f3208d15640 1 -- 172.21.15.19:0/1692387152 >> 172.21.15.19:0/1692387152 conn(0x7f32040b4e90 msgr2=0x7f32040b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.611+0000 7f3208d15640 1 -- 172.21.15.19:0/1692387152 shutdown_connections 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.611+0000 7f3208d15640 1 -- 172.21.15.19:0/1692387152 wait complete. 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.612+0000 7f3208d15640 10 monclient: build_initial_monmap 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.612+0000 7f3208d15640 10 monclient: monmap: 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:53.618 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.612+0000 7f3208d15640 1 librados: starting msgr at 2024-05-04T18:18:53.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.612+0000 7f3208d15640 1 librados: starting objecter 2024-05-04T18:18:53.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.612+0000 7f3208d15640 1 Processor -- start 2024-05-04T18:18:53.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.612+0000 7f3208d15640 1 -- start start 2024-05-04T18:18:53.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.612+0000 7f3208d15640 1 librados: setting wanted keys 2024-05-04T18:18:53.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.612+0000 7f3208d15640 1 librados: calling monclient init 2024-05-04T18:18:53.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.612+0000 7f3208d15640 10 monclient: init 2024-05-04T18:18:53.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:53.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-04T18:18:53.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f32040a9fb0 0x7f3204142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:53.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 10 monclient(hunting): picked mon.noname-c con 0x7f32040a9fb0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:53.619 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32040b27f0 0x7f3204132820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:53.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 10 monclient(hunting): picked mon.noname-a con 0x7f32040b27f0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:53.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32041404e0 0x7f3204134260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:53.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 10 monclient(hunting): picked mon.noname-b con 0x7f32041404e0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:53.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:53.620 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f320414e7b0 con 0x7f32040b27f0 2024-05-04T18:18:53.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:53.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f320414e630 con 0x7f32041404e0 2024-05-04T18:18:53.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:53.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f320414e930 con 0x7f32040a9fb0 2024-05-04T18:18:53.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:53.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3208d15640 10 monclient(hunting): authenticate will time out at 3355.728112s 2024-05-04T18:18:53.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f3201d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32040b27f0 0x7f3204132820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:53.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.613+0000 7f320259c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f32040a9fb0 0x7f3204142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:53.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3202d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32041404e0 0x7f3204134260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3202d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32041404e0 0x7f3204134260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:45312/0 (socket says 172.21.15.19:45312) 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3202d9d640 1 -- 172.21.15.19:0/3601147181 learned_addr learned my addr 172.21.15.19:0/3601147181 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3202d9d640 10 monclient(hunting): get_auth_request con 0x7f32041404e0 auth_method 0 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3202d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3202d9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3202d9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f320259c640 10 monclient(hunting): get_auth_request con 0x7f32040a9fb0 auth_method 0 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f320259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f320259c640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f320259c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3201d9b640 10 monclient(hunting): get_auth_request con 0x7f32040b27f0 auth_method 0 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3201d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:53.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3201d9b640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:53.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3201d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:53.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3202d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:53.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3202d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:53.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.614+0000 7f3202d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:53.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.615+0000 7f320259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:53.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.615+0000 7f320259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:53.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.615+0000 7f320259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:53.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.615+0000 7f3201d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:53.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.615+0000 7f3201d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:53.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.615+0000 7f3201d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:53.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.617+0000 7f3202d9d640 10 monclient(hunting): handle_auth_done global_id 22741 payload 824 2024-05-04T18:18:53.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.617+0000 7f3202d9d640 1 -- 172.21.15.19:0/3601147181 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f32040a9fb0 msgr2=0x7f3204142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.617+0000 7f3202d9d640 1 --2- 172.21.15.19:0/3601147181 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f32040a9fb0 0x7f3204142fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.617+0000 7f3202d9d640 1 -- 172.21.15.19:0/3601147181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32040b27f0 msgr2=0x7f3204132820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.617+0000 7f3202d9d640 1 --2- 172.21.15.19:0/3601147181 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f32040b27f0 0x7f3204132820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.617+0000 7f3202d9d640 10 monclient: _finish_hunting 0 2024-05-04T18:18:53.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.617+0000 7f3202d9d640 1 monclient: found mon.noname-b 2024-05-04T18:18:53.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.617+0000 7f3202d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:53.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.617+0000 7f3202d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:53.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.617+0000 7f3202d9d640 1 -- 172.21.15.19:0/3601147181 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3204133bd0 con 0x7f32041404e0 2024-05-04T18:18:53.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.620+0000 7f3202d9d640 1 --2- 172.21.15.19:0/3601147181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32041404e0 0x7f3204134260 secure :-1 s=READY pgs=5824 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00c760 tx=0x7f31fc00cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:53.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.620+0000 7f31f2ffd640 10 client.?.objecter ms_handle_connect 0x7f32041404e0 2024-05-04T18:18:53.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.620+0000 7f31f2ffd640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:53.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 1 -- 172.21.15.19:0/3601147181 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc003b70 con 0x7f32041404e0 2024-05-04T18:18:53.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:53.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:53.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 10 monclient: dump: 2024-05-04T18:18:53.626 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:53.626 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 10 monclient: _finish_auth 0 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 10 monclient: _check_auth_tickets 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 1 -- 172.21.15.19:0/3601147181 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31fc003d10 con 0x7f32041404e0 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 1 -- 172.21.15.19:0/3601147181 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc00aa90 con 0x7f32041404e0 2024-05-04T18:18:53.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f31f2ffd640 10 monclient: dump: 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 5 monclient: authenticate success, global_id 22741 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 10 monclient: _renew_subs 2024-05-04T18:18:53.628 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3200d99640 4 set_mon_vals no callback set 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 1 -- 172.21.15.19:0/3601147181 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3204133e40 con 0x7f32041404e0 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 10 client.22741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 10 monclient: _renew_subs 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 1 -- 172.21.15.19:0/3601147181 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3204134980 con 0x7f32041404e0 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 1 librados: init done 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 10 client.22741.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 15 client.22741.objecter _session_command_op_assign -1 1 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 20 client.22741.objecter _get_session osd=-1 returning homeless 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 10 client.22741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:53.629 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.624+0000 7f3208d15640 1 -- 172.21.15.19:0/3601147181 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f32040b4bd0 con 0x7f32041404e0 2024-05-04T18:18:53.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.645+0000 7f31f2ffd640 1 -- 172.21.15.19:0/3601147181 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f31fc00ac30 con 0x7f32041404e0 2024-05-04T18:18:53.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.645+0000 7f31f2ffd640 1 --2- 172.21.15.19:0/3601147181 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f31e406d810 0x7f31e406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:53.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.646+0000 7f320259c640 1 --2- 172.21.15.19:0/3601147181 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f31e406d810 0x7f31e406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:53.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.646+0000 7f320259c640 10 monclient: get_auth_request con 0x7f31e406d810 auth_method 0 2024-05-04T18:18:53.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.646+0000 7f320259c640 1 --2- 172.21.15.19:0/3601147181 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f31e406d810 0x7f31e406fcd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f31f4036000 tx=0x7f31f403c040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:53.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.646+0000 7f31f2ffd640 10 client.22741.objecter ms_handle_connect 0x7f31e406d810 2024-05-04T18:18:53.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 1 -- 172.21.15.19:0/3601147181 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f31fc057a00 con 0x7f32041404e0 2024-05-04T18:18:53.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 10 client.22741.objecter ms_dispatch 0x7f3204000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:53.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 3 client.22741.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:53.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 3 client.22741.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:53.651 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 10 client.22741.objecter checking command 1 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 1 --2- 172.21.15.19:0/3601147181 >> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] conn(0x7f31e40756d0 0x7f31e4077b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 20 client.22741.objecter _get_session s=0x7f31e4075000 osd=1 3 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 20 client.22741.objecter put_session s=0x7f31e4075000 osd=1 3 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 15 client.22741.objecter _session_command_op_remove -1 1 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 20 client.22741.objecter _get_session s=0x7f31e4075000 osd=1 3 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 20 client.22741.objecter get_session s=0x7f31e4075000 osd=1 3 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 15 client.22741.objecter _session_command_op_assign 1 1 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 20 client.22741.objecter put_session s=0x7f31e4075000 osd=1 4 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 10 client.22741.objecter _send_command 1 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 1 -- 172.21.15.19:0/3601147181 --> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31e4078240 con 0x7f31e40756d0 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 20 client.22741.objecter dump_active .. 0 homeless 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 1 -- 172.21.15.19:0/3601147181 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=1186) ==== 24+0+0 (secure 0 0 0) 0x7f31fc055c00 con 0x7f32041404e0 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.649+0000 7f31f2ffd640 10 monclient: handle_get_version_reply finishing 1 version 1186 2024-05-04T18:18:53.652 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.650+0000 7f3201d9b640 1 --2- 172.21.15.19:0/3601147181 >> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] conn(0x7f31e40756d0 0x7f31e4077b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:53.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.650+0000 7f3201d9b640 10 monclient: get_auth_request con 0x7f31e40756d0 auth_method 0 2024-05-04T18:18:53.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.653+0000 7f31f2ffd640 10 client.22741.objecter ms_handle_connect 0x7f31e40756d0 2024-05-04T18:18:53.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.653+0000 7f3201d9b640 1 --2- 172.21.15.19:0/3601147181 >> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] conn(0x7f31e40756d0 0x7f31e4077b70 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.677+0000 7f31f2ffd640 1 -- 172.21.15.19:0/3601147181 <== osd.1 v2:172.21.15.19:6800/3360656308 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31e4078240 con 0x7f31e40756d0 2024-05-04T18:18:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.677+0000 7f31f2ffd640 10 client.22741.objecter ms_dispatch 0x7f3204000b90 command_reply(tid 1: 0 ) 2024-05-04T18:18:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.677+0000 7f31f2ffd640 10 client.22741.objecter _finish_command 1 = system:0 2024-05-04T18:18:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.677+0000 7f31f2ffd640 20 client.22741.objecter put_session s=0x7f31e4075000 osd=1 4 2024-05-04T18:18:53.679 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.677+0000 7f31f2ffd640 15 client.22741.objecter _session_command_op_remove 1 1 2024-05-04T18:18:53.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.692+0000 7f3208d15640 10 client.22741.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-04T18:18:53.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.692+0000 7f3208d15640 15 client.22741.objecter _session_command_op_assign -1 2 2024-05-04T18:18:53.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.692+0000 7f3208d15640 20 client.22741.objecter _get_session s=0x7f31e4075000 osd=1 3 2024-05-04T18:18:53.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.692+0000 7f3208d15640 20 client.22741.objecter put_session s=0x7f31e4075000 osd=1 3 2024-05-04T18:18:53.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.692+0000 7f3208d15640 20 client.22741.objecter _get_session s=0x7f31e4075000 osd=1 3 2024-05-04T18:18:53.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.692+0000 7f3208d15640 15 client.22741.objecter _session_command_op_remove -1 2 2024-05-04T18:18:53.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.692+0000 7f3208d15640 20 client.22741.objecter get_session s=0x7f31e4075000 osd=1 3 2024-05-04T18:18:53.693 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.692+0000 7f3208d15640 15 client.22741.objecter _session_command_op_assign 1 2 2024-05-04T18:18:53.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.692+0000 7f3208d15640 20 client.22741.objecter put_session s=0x7f31e4075000 osd=1 4 2024-05-04T18:18:53.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.692+0000 7f3208d15640 10 client.22741.objecter _send_command 2 2024-05-04T18:18:53.694 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.692+0000 7f3208d15640 1 -- 172.21.15.19:0/3601147181 --> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3204068b10 con 0x7f31e40756d0 2024-05-04T18:18:53.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f31f2ffd640 1 -- 172.21.15.19:0/3601147181 <== osd.1 v2:172.21.15.19:6800/3360656308 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3204068b10 con 0x7f31e40756d0 2024-05-04T18:18:53.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f31f2ffd640 10 client.22741.objecter ms_dispatch 0x7f3204000b90 command_reply(tid 2: 0 ) 2024-05-04T18:18:53.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f31f2ffd640 10 client.22741.objecter _finish_command 2 = system:0 2024-05-04T18:18:53.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f31f2ffd640 20 client.22741.objecter put_session s=0x7f31e4075000 osd=1 4 2024-05-04T18:18:53.709 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f31f2ffd640 15 client.22741.objecter _session_command_op_remove 1 2 2024-05-04T18:18:53.710 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-05-04T18:18:53.710 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T18:18:53.710 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-05-04T18:18:53.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 10 librados: watch_flush enter 2024-05-04T18:18:53.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 10 librados: watch_flush exit 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 10 client.22741.objecter close_session for osd.1 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 1 -- 172.21.15.19:0/3601147181 >> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] conn(0x7f31e40756d0 msgr2=0x7f31e4077b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 1 --2- 172.21.15.19:0/3601147181 >> [v2:172.21.15.19:6800/3360656308,v1:172.21.15.19:6801/3360656308] conn(0x7f31e40756d0 0x7f31e4077b70 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 20 client.22741.objecter put_session s=0x7f31e4075000 osd=1 1 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 20 client.22741.objecter shutdown clearing up homeless session... 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 10 client.22741.objecter successfully canceled tick 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 1 -- 172.21.15.19:0/3601147181 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f31e406d810 msgr2=0x7f31e406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 1 --2- 172.21.15.19:0/3601147181 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f31e406d810 0x7f31e406fcd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f31f4036000 tx=0x7f31f403c040 comp rx=0 tx=0).stop 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 10 monclient: shutdown 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 1 -- 172.21.15.19:0/3601147181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32041404e0 msgr2=0x7f3204134260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 1 --2- 172.21.15.19:0/3601147181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32041404e0 0x7f3204134260 secure :-1 s=READY pgs=5824 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00c760 tx=0x7f31fc00cc20 comp rx=0 tx=0).stop 2024-05-04T18:18:53.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3202d9d640 1 -- 172.21.15.19:0/3601147181 reap_dead start 2024-05-04T18:18:53.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 1 -- 172.21.15.19:0/3601147181 shutdown_connections 2024-05-04T18:18:53.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 1 -- 172.21.15.19:0/3601147181 >> 172.21.15.19:0/3601147181 conn(0x7f32040b4e90 msgr2=0x7f32040b5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:53.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 1 -- 172.21.15.19:0/3601147181 shutdown_connections 2024-05-04T18:18:53.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 1 -- 172.21.15.19:0/3601147181 wait complete. 2024-05-04T18:18:53.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.708+0000 7f3208d15640 1 librados: shutdown 2024-05-04T18:18:53.720 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-04T18:18:53.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.821+0000 7f017aa0d640 10 monclient: get_monmap_and_config 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.821+0000 7f017aa0d640 10 monclient: build_initial_monmap 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.821+0000 7f017aa0d640 10 monclient: monmap: 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:53.823740+0000 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:53.823740+0000 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:53.824 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:53.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.822+0000 7f017aa0d640 1 Processor -- start 2024-05-04T18:18:53.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.822+0000 7f017aa0d640 1 -- start start 2024-05-04T18:18:53.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.822+0000 7f017aa0d640 10 monclient: init 2024-05-04T18:18:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f017aa0d640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f017aa0d640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-05-04T18:18:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f017aa0d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01740b2690 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f017aa0d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f017413fd70 0x7f01741401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f017aa0d640 10 monclient(hunting): picked mon.noname-c con 0x7f017413fd70 addr v2:172.21.15.186:3300/0 2024-05-04T18:18:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f017aa0d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0174142c70 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f017aa0d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01741406f0 con 0x7f0174142c70 2024-05-04T18:18:53.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f017aa0d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01741437b0 con 0x7f01740b2690 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f017aa0d640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f017aa0d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f01740b2a70 con 0x7f017413fd70 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f017aa0d640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f017aa0d640 10 monclient(hunting): authenticate will time out at 3355.937985s 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.823+0000 7f0178faa640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f0174142c70 0x7f01741430a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:53086/0 (socket says 172.21.15.19:53086) 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.824+0000 7f0178faa640 1 -- 172.21.15.19:0/4234033743 learned_addr learned my addr 172.21.15.19:0/4234033743 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.824+0000 7f01737fe640 1 --2- 172.21.15.19:0/4234033743 >> v2:172.21.15.186:3300/0 conn(0x7f017413fd70 0x7f01741401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.824+0000 7f01737fe640 10 monclient(hunting): get_auth_request con 0x7f017413fd70 auth_method 0 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.824+0000 7f01737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.824+0000 7f01737fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.824+0000 7f01737fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.825+0000 7f01737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.825+0000 7f01737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:53.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.825+0000 7f01737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:53.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f0172ffd640 1 -- 172.21.15.19:0/4234033743 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3539815199 0 0) 0x7f01741406f0 con 0x7f0174142c70 2024-05-04T18:18:53.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f0172ffd640 10 monclient(hunting): _negotiate 2024-05-04T18:18:53.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f0172ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:53.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f0172ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:53.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f0172ffd640 10 monclient(hunting): my global_id is 15777 2024-05-04T18:18:53.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f0172ffd640 1 -- 172.21.15.19:0/4234033743 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0160003520 con 0x7f0174142c70 2024-05-04T18:18:53.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f01737fe640 10 monclient(hunting): handle_auth_done global_id 22744 payload 274 2024-05-04T18:18:53.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f01737fe640 1 -- 172.21.15.19:0/4234033743 >> v1:172.21.15.186:6789/0 conn(0x7f01740b2690 legacy=0x7f01740b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:53.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f01737fe640 1 -- 172.21.15.19:0/4234033743 >> v1:172.21.15.19:6789/0 conn(0x7f0174142c70 legacy=0x7f01741430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f01737fe640 10 monclient: _finish_hunting 0 2024-05-04T18:18:53.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f01737fe640 1 monclient: found mon.noname-c 2024-05-04T18:18:53.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f01737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:53.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f01737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T18:18:53.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.826+0000 7f01737fe640 1 -- 172.21.15.19:0/4234033743 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0174073680 con 0x7f017413fd70 2024-05-04T18:18:53.830 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.829+0000 7f01737fe640 1 --2- 172.21.15.19:0/4234033743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f017413fd70 0x7f01741401b0 secure :-1 s=READY pgs=5825 cs=0 l=1 rev1=1 crypto rx=0x7f015c0097a0 tx=0x7f015c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=691b3d1ed227fa5f server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:53.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 1 -- 172.21.15.19:0/4234033743 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c0315a0 con 0x7f017413fd70 2024-05-04T18:18:53.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:53.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 10 monclient: dump: 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 10 monclient: _finish_auth 0 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 10 monclient: _check_auth_tickets 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 1 -- 172.21.15.19:0/4234033743 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f015c037070 con 0x7f017413fd70 2024-05-04T18:18:53.835 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f017aa0d640 5 monclient: authenticate success, global_id 22744 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f017aa0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f017aa0d640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 1 -- 172.21.15.19:0/4234033743 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015c031960 con 0x7f017413fd70 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f017aa0d640 4 set_mon_vals no callback set 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f0172ffd640 10 monclient: dump: 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:53.836 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:53.837 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:53.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f017aa0d640 10 monclient: shutdown 2024-05-04T18:18:53.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f017aa0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:53.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f017aa0d640 1 -- 172.21.15.19:0/4234033743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f017413fd70 msgr2=0x7f01741401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.833+0000 7f017aa0d640 1 --2- 172.21.15.19:0/4234033743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f017413fd70 0x7f01741401b0 secure :-1 s=READY pgs=5825 cs=0 l=1 rev1=1 crypto rx=0x7f015c0097a0 tx=0x7f015c02b840 comp rx=0 tx=0).stop 2024-05-04T18:18:53.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.834+0000 7f017aa0d640 1 -- 172.21.15.19:0/4234033743 shutdown_connections 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.834+0000 7f017aa0d640 1 --2- 172.21.15.19:0/4234033743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f017413fd70 0x7f01741401b0 unknown :-1 s=CLOSED pgs=5825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.834+0000 7f017aa0d640 1 -- 172.21.15.19:0/4234033743 >> 172.21.15.19:0/4234033743 conn(0x7f01740b5840 msgr2=0x7f01740b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.834+0000 7f017aa0d640 1 -- 172.21.15.19:0/4234033743 shutdown_connections 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.834+0000 7f017aa0d640 1 -- 172.21.15.19:0/4234033743 wait complete. 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.834+0000 7f017aa0d640 10 monclient: build_initial_monmap 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.834+0000 7f017aa0d640 10 monclient: monmap: 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:53.843 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:53.844 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:53.844 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:53.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.834+0000 7f017aa0d640 1 librados: starting msgr at 2024-05-04T18:18:53.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.834+0000 7f017aa0d640 1 librados: starting objecter 2024-05-04T18:18:53.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 1 Processor -- start 2024-05-04T18:18:53.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 1 -- start start 2024-05-04T18:18:53.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 1 librados: setting wanted keys 2024-05-04T18:18:53.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 1 librados: calling monclient init 2024-05-04T18:18:53.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 10 monclient: init 2024-05-04T18:18:53.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:53.844 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f01740b2690 0x7f0174143fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 10 monclient(hunting): picked mon.noname-c con 0x7f01740b2690 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f017413fd70 0x7f017406fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 10 monclient(hunting): picked mon.noname-a con 0x7f017413fd70 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0174142c70 0x7f01741db670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 10 monclient(hunting): picked mon.noname-b con 0x7f0174142c70 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f01740b1a20 con 0x7f017413fd70 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0174143980 con 0x7f0174142c70 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f017414e5b0 con 0x7f01740b2690 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:53.845 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.835+0000 7f017aa0d640 10 monclient(hunting): authenticate will time out at 3355.950550s 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f0173fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f01740b2690 0x7f0174143fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f0178faa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0174142c70 0x7f01741db670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f01737fe640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f017413fd70 0x7f017406fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f0173fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f01740b2690 0x7f0174143fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58124/0 (socket says 172.21.15.19:58124) 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f0173fff640 1 -- 172.21.15.19:0/494088065 learned_addr learned my addr 172.21.15.19:0/494088065 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f0173fff640 10 monclient(hunting): get_auth_request con 0x7f01740b2690 auth_method 0 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f0173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f0173fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f0173fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f01737fe640 10 monclient(hunting): get_auth_request con 0x7f017413fd70 auth_method 0 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f01737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f01737fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f01737fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:53.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f0178faa640 10 monclient(hunting): get_auth_request con 0x7f0174142c70 auth_method 0 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f0178faa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f0178faa640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.836+0000 7f0178faa640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.837+0000 7f0173fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.837+0000 7f0173fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.837+0000 7f0173fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.837+0000 7f0178faa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.837+0000 7f0178faa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.837+0000 7f0178faa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.837+0000 7f01737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.837+0000 7f01737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.837+0000 7f01737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.840+0000 7f0178faa640 10 monclient(hunting): handle_auth_done global_id 22747 payload 824 2024-05-04T18:18:53.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.840+0000 7f0178faa640 1 -- 172.21.15.19:0/494088065 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f01740b2690 msgr2=0x7f0174143fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.840+0000 7f0178faa640 1 --2- 172.21.15.19:0/494088065 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f01740b2690 0x7f0174143fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.840+0000 7f0178faa640 1 -- 172.21.15.19:0/494088065 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f017413fd70 msgr2=0x7f017406fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:53.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.840+0000 7f0178faa640 1 --2- 172.21.15.19:0/494088065 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f017413fd70 0x7f017406fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:53.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.840+0000 7f0178faa640 10 monclient: _finish_hunting 0 2024-05-04T18:18:53.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.840+0000 7f0178faa640 1 monclient: found mon.noname-b 2024-05-04T18:18:53.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.840+0000 7f0178faa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:53.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.840+0000 7f0178faa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:53.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.840+0000 7f0178faa640 1 -- 172.21.15.19:0/494088065 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0174144ff0 con 0x7f0174142c70 2024-05-04T18:18:53.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.842+0000 7f0178faa640 1 --2- 172.21.15.19:0/494088065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0174142c70 0x7f01741db670 secure :-1 s=READY pgs=5826 cs=0 l=1 rev1=1 crypto rx=0x7f016c00caa0 tx=0x7f016c00cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:53.849 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.842+0000 7f0170ff9640 10 client.?.objecter ms_handle_connect 0x7f0174142c70 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.842+0000 7f0170ff9640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 1 -- 172.21.15.19:0/494088065 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c017070 con 0x7f0174142c70 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 10 monclient: dump: 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:53.850 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 10 monclient: _finish_auth 0 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 10 monclient: _check_auth_tickets 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 1 -- 172.21.15.19:0/494088065 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f016c00aae0 con 0x7f0174142c70 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 1 -- 172.21.15.19:0/494088065 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c012400 con 0x7f0174142c70 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f017aa0d640 5 monclient: authenticate success, global_id 22747 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f017aa0d640 10 monclient: _renew_subs 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f017aa0d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f017aa0d640 1 -- 172.21.15.19:0/494088065 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01741452d0 con 0x7f0174142c70 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0171ffb640 4 set_mon_vals no callback set 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f0170ff9640 10 monclient: dump: 2024-05-04T18:18:53.851 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f017aa0d640 10 client.22747.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f017aa0d640 10 monclient: _renew_subs 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f017aa0d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f017aa0d640 1 -- 172.21.15.19:0/494088065 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01740721f0 con 0x7f0174142c70 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.846+0000 7f017aa0d640 1 librados: init done 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.847+0000 7f017aa0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:53.852 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.847+0000 7f017aa0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:53.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.847+0000 7f017aa0d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:53.853 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:53.847+0000 7f017aa0d640 1 -- 172.21.15.19:0/494088065 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01741456c0 con 0x7f0174142c70 2024-05-04T18:18:54.098 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.097+0000 7f0170ff9640 1 -- 172.21.15.19:0/494088065 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f016c01a070 con 0x7f0174142c70 2024-05-04T18:18:54.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.097+0000 7f0170ff9640 1 --2- 172.21.15.19:0/494088065 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f015406d810 0x7f015406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.097+0000 7f0173fff640 1 --2- 172.21.15.19:0/494088065 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f015406d810 0x7f015406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.097+0000 7f0173fff640 10 monclient: get_auth_request con 0x7f015406d810 auth_method 0 2024-05-04T18:18:54.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.098+0000 7f0173fff640 1 --2- 172.21.15.19:0/494088065 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f015406d810 0x7f015406fcd0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f01640018b0 tx=0x7f0164009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:54.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.098+0000 7f0170ff9640 10 client.22747.objecter ms_handle_connect 0x7f015406d810 2024-05-04T18:18:54.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.100+0000 7f0170ff9640 1 -- 172.21.15.19:0/494088065 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f016c0572c0 con 0x7f0174142c70 2024-05-04T18:18:54.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.100+0000 7f0170ff9640 10 client.22747.objecter ms_dispatch 0x7f0174000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:54.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.100+0000 7f0170ff9640 3 client.22747.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:54.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.100+0000 7f0170ff9640 3 client.22747.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:54.102 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.101+0000 7f0170ff9640 20 client.22747.objecter dump_active .. 0 homeless 2024-05-04T18:18:54.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.111+0000 7f0170ff9640 1 -- 172.21.15.19:0/494088065 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f016c059220 con 0x7f0174142c70 2024-05-04T18:18:54.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.111+0000 7f0170ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:54.112 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.111+0000 7f0170ff9640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:18:54.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.298+0000 7f017aa0d640 1 -- 172.21.15.19:0/494088065 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f01741454a0 con 0x7f015406d810 2024-05-04T18:18:54.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.299+0000 7f0170ff9640 1 -- 172.21.15.19:0/494088065 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f01741454a0 con 0x7f015406d810 2024-05-04T18:18:54.300 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 1 to deep-scrub 2024-05-04T18:18:54.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 10 librados: watch_flush enter 2024-05-04T18:18:54.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 10 librados: watch_flush exit 2024-05-04T18:18:54.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 20 client.22747.objecter shutdown clearing up homeless session... 2024-05-04T18:18:54.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 10 client.22747.objecter successfully canceled tick 2024-05-04T18:18:54.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 -- 172.21.15.19:0/494088065 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f015406d810 msgr2=0x7f015406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 --2- 172.21.15.19:0/494088065 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f015406d810 0x7f015406fcd0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f01640018b0 tx=0x7f0164009040 comp rx=0 tx=0).stop 2024-05-04T18:18:54.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 10 monclient: shutdown 2024-05-04T18:18:54.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:54.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 -- 172.21.15.19:0/494088065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0174142c70 msgr2=0x7f01741db670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 --2- 172.21.15.19:0/494088065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0174142c70 0x7f01741db670 secure :-1 s=READY pgs=5826 cs=0 l=1 rev1=1 crypto rx=0x7f016c00caa0 tx=0x7f016c00cea0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 -- 172.21.15.19:0/494088065 shutdown_connections 2024-05-04T18:18:54.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 --2- 172.21.15.19:0/494088065 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f015406d810 0x7f015406fcd0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 --2- 172.21.15.19:0/494088065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0174142c70 0x7f01741db670 unknown :-1 s=CLOSED pgs=5826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 --2- 172.21.15.19:0/494088065 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f017413fd70 0x7f017406fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 --2- 172.21.15.19:0/494088065 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f01740b2690 0x7f0174143fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 -- 172.21.15.19:0/494088065 >> 172.21.15.19:0/494088065 conn(0x7f01740b5840 msgr2=0x7f017413ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:54.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 -- 172.21.15.19:0/494088065 shutdown_connections 2024-05-04T18:18:54.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 -- 172.21.15.19:0/494088065 wait complete. 2024-05-04T18:18:54.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.302+0000 7f017aa0d640 1 librados: shutdown 2024-05-04T18:18:54.315 INFO:tasks.ceph:Scrubbing osd.2 2024-05-04T18:18:54.316 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-04T18:18:54.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.419+0000 7f9f40e0e640 10 monclient: get_monmap_and_config 2024-05-04T18:18:54.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.419+0000 7f9f40e0e640 10 monclient: build_initial_monmap 2024-05-04T18:18:54.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.419+0000 7f9f40e0e640 10 monclient: monmap: 2024-05-04T18:18:54.420 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:54.420 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:54.420 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:54.420964+0000 2024-05-04T18:18:54.421 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:54.420964+0000 2024-05-04T18:18:54.421 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:54.421 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:54.421 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:54.421 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:54.421 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:54.421 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:54.421 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:54.421 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.419+0000 7f9f40e0e640 1 Processor -- start 2024-05-04T18:18:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.419+0000 7f9f40e0e640 1 -- start start 2024-05-04T18:18:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.419+0000 7f9f40e0e640 10 monclient: init 2024-05-04T18:18:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-05-04T18:18:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f9f3c0a9fb0 0x7f9f3c0aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 10 monclient(hunting): picked mon.noname-a con 0x7f9f3c0a9fb0 addr v2:172.21.15.19:3300/0 2024-05-04T18:18:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0b27f0 0x7f9f3c13fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 10 monclient(hunting): picked mon.noname-b con 0x7f9f3c0b27f0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9f3c140310 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f3c0ab880 con 0x7f9f3c140310 2024-05-04T18:18:54.423 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f9f3c1407a0 con 0x7f9f3c0a9fb0 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f9f3c146b00 con 0x7f9f3c0b27f0 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.420+0000 7f9f40e0e640 10 monclient(hunting): authenticate will time out at 3356.535224s 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.421+0000 7f9f33fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0b27f0 0x7f9f3c13fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.421+0000 7f9f3a59c640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f9f3c0a9fb0 0x7f9f3c0aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.421+0000 7f9f33fff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0b27f0 0x7f9f3c13fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58126/0 (socket says 172.21.15.19:58126) 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.421+0000 7f9f33fff640 1 -- 172.21.15.19:0/993285892 learned_addr learned my addr 172.21.15.19:0/993285892 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.421+0000 7f9f33fff640 10 monclient(hunting): get_auth_request con 0x7f9f3c0b27f0 auth_method 0 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.421+0000 7f9f33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.421+0000 7f9f33fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.421+0000 7f9f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.421+0000 7f9f3a59c640 10 monclient(hunting): get_auth_request con 0x7f9f3c0a9fb0 auth_method 0 2024-05-04T18:18:54.424 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.421+0000 7f9f3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:54.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.421+0000 7f9f3a59c640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:54.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.421+0000 7f9f3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:54.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.422+0000 7f9f33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:54.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.422+0000 7f9f33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:54.425 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.422+0000 7f9f33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.422+0000 7f9f3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.422+0000 7f9f3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.422+0000 7f9f3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.423+0000 7f9f33fff640 10 monclient(hunting): handle_auth_done global_id 18794 payload 274 2024-05-04T18:18:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.423+0000 7f9f33fff640 1 -- 172.21.15.19:0/993285892 >> v2:172.21.15.19:3300/0 conn(0x7f9f3c0a9fb0 msgr2=0x7f9f3c0aa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.423+0000 7f9f33fff640 1 --2- 172.21.15.19:0/993285892 >> v2:172.21.15.19:3300/0 conn(0x7f9f3c0a9fb0 0x7f9f3c0aa390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.423+0000 7f9f33fff640 1 -- 172.21.15.19:0/993285892 >> v1:172.21.15.19:6789/0 conn(0x7f9f3c140310 legacy=0x7f9f3c1463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.423+0000 7f9f33fff640 10 monclient: _finish_hunting 0 2024-05-04T18:18:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.423+0000 7f9f33fff640 1 monclient: found mon.noname-b 2024-05-04T18:18:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.423+0000 7f9f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.423+0000 7f9f33fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T18:18:54.426 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.423+0000 7f9f33fff640 1 -- 172.21.15.19:0/993285892 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f3c146cb0 con 0x7f9f3c0b27f0 2024-05-04T18:18:54.427 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.425+0000 7f9f33fff640 1 --2- 172.21.15.19:0/993285892 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0b27f0 0x7f9f3c13fdd0 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 crypto rx=0x7f9f24009870 tx=0x7f9f2402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22c5e2d85048c167 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:54.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 1 -- 172.21.15.19:0/993285892 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f24035020 con 0x7f9f3c0b27f0 2024-05-04T18:18:54.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:54.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:54.431 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 10 monclient: dump: 2024-05-04T18:18:54.431 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:54.431 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:54.431 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.431 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.431 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:54.431 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 10 monclient: _finish_auth 0 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 10 monclient: _check_auth_tickets 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 1 -- 172.21.15.19:0/993285892 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f24031640 con 0x7f9f3c0b27f0 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f40e0e640 5 monclient: authenticate success, global_id 18794 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f40e0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f40e0e640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f40e0e640 4 set_mon_vals no callback set 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 1 -- 172.21.15.19:0/993285892 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2402d040 con 0x7f9f3c0b27f0 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:54.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:54.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.429+0000 7f9f39d9b640 10 monclient: dump: 2024-05-04T18:18:54.433 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:54.433 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:54.433 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.433 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.433 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:54.433 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:54.433 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:54.433 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:54.433 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:54.433 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:54.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.430+0000 7f9f40e0e640 10 monclient: shutdown 2024-05-04T18:18:54.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.430+0000 7f9f40e0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:54.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.430+0000 7f9f40e0e640 1 -- 172.21.15.19:0/993285892 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0b27f0 msgr2=0x7f9f3c13fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.430+0000 7f9f40e0e640 1 --2- 172.21.15.19:0/993285892 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0b27f0 0x7f9f3c13fdd0 secure :-1 s=READY pgs=3165 cs=0 l=1 rev1=1 crypto rx=0x7f9f24009870 tx=0x7f9f2402b8f0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.430+0000 7f9f40e0e640 1 -- 172.21.15.19:0/993285892 shutdown_connections 2024-05-04T18:18:54.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.430+0000 7f9f40e0e640 1 --2- 172.21.15.19:0/993285892 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0b27f0 0x7f9f3c13fdd0 unknown :-1 s=CLOSED pgs=3165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.430+0000 7f9f40e0e640 1 --2- 172.21.15.19:0/993285892 >> v2:172.21.15.19:3300/0 conn(0x7f9f3c0a9fb0 0x7f9f3c0aa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.430+0000 7f9f40e0e640 1 -- 172.21.15.19:0/993285892 >> 172.21.15.19:0/993285892 conn(0x7f9f3c0b4e90 msgr2=0x7f9f3c0b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.430+0000 7f9f40e0e640 1 -- 172.21.15.19:0/993285892 shutdown_connections 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.430+0000 7f9f40e0e640 1 -- 172.21.15.19:0/993285892 wait complete. 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.431+0000 7f9f40e0e640 10 monclient: build_initial_monmap 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.431+0000 7f9f40e0e640 10 monclient: monmap: 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:54.437 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.431+0000 7f9f40e0e640 1 librados: starting msgr at 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.431+0000 7f9f40e0e640 1 librados: starting objecter 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.431+0000 7f9f40e0e640 1 Processor -- start 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 1 -- start start 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 1 librados: setting wanted keys 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 1 librados: calling monclient init 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 10 monclient: init 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0a9fb0 0x7f9f3c13a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 10 monclient(hunting): picked mon.noname-c con 0x7f9f3c0a9fb0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f3c0b27f0 0x7f9f3c137260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 10 monclient(hunting): picked mon.noname-b con 0x7f9f3c0b27f0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:54.438 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9f3c140310 0x7f9f3c134510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 10 monclient(hunting): picked mon.noname-a con 0x7f9f3c140310 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3c14d7a0 con 0x7f9f3c140310 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3c14d620 con 0x7f9f3c0b27f0 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f9f3c14d920 con 0x7f9f3c0a9fb0 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.432+0000 7f9f40e0e640 10 monclient(hunting): authenticate will time out at 3356.547286s 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.433+0000 7f9f3a59c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0a9fb0 0x7f9f3c13a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.433+0000 7f9f3a59c640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0a9fb0 0x7f9f3c13a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58136/0 (socket says 172.21.15.19:58136) 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.433+0000 7f9f3a59c640 1 -- 172.21.15.19:0/3604367410 learned_addr learned my addr 172.21.15.19:0/3604367410 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.433+0000 7f9f3a59c640 10 monclient(hunting): get_auth_request con 0x7f9f3c0a9fb0 auth_method 0 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.433+0000 7f9f3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:54.439 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.433+0000 7f9f3a59c640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.433+0000 7f9f3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.433+0000 7f9f3ad9d640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9f3c140310 0x7f9f3c134510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.433+0000 7f9f3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.433+0000 7f9f3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.433+0000 7f9f3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.434+0000 7f9f3ad9d640 10 monclient(hunting): get_auth_request con 0x7f9f3c140310 auth_method 0 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.434+0000 7f9f3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.434+0000 7f9f3ad9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.434+0000 7f9f3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.434+0000 7f9f3ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.434+0000 7f9f3ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.434+0000 7f9f3ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:54.440 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.434+0000 7f9f33fff640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f3c0b27f0 0x7f9f3c137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.435+0000 7f9f33fff640 10 monclient(hunting): get_auth_request con 0x7f9f3c0b27f0 auth_method 0 2024-05-04T18:18:54.441 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.435+0000 7f9f33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.435+0000 7f9f33fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.435+0000 7f9f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.436+0000 7f9f33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.436+0000 7f9f33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.436+0000 7f9f33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.437+0000 7f9f3a59c640 10 monclient(hunting): handle_auth_done global_id 18797 payload 824 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.437+0000 7f9f3a59c640 1 -- 172.21.15.19:0/3604367410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f3c0b27f0 msgr2=0x7f9f3c137260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.437+0000 7f9f3a59c640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f3c0b27f0 0x7f9f3c137260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.437+0000 7f9f3a59c640 1 -- 172.21.15.19:0/3604367410 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9f3c140310 msgr2=0x7f9f3c134510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.437+0000 7f9f3a59c640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9f3c140310 0x7f9f3c134510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.437+0000 7f9f3a59c640 10 monclient: _finish_hunting 0 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.437+0000 7f9f3a59c640 1 monclient: found mon.noname-c 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.437+0000 7f9f3a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.437+0000 7f9f3a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T18:18:54.442 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.437+0000 7f9f3a59c640 1 -- 172.21.15.19:0/3604367410 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f3c1377a0 con 0x7f9f3c0a9fb0 2024-05-04T18:18:54.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.439+0000 7f9f3a59c640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0a9fb0 0x7f9f3c13a4b0 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c00e8a0 tx=0x7f9f2c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:54.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.439+0000 7f9f32ffd640 10 client.?.objecter ms_handle_connect 0x7f9f3c0a9fb0 2024-05-04T18:18:54.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.439+0000 7f9f32ffd640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:54.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 1 -- 172.21.15.19:0/3604367410 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c017070 con 0x7f9f3c0a9fb0 2024-05-04T18:18:54.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:54.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T18:18:54.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 10 monclient: dump: 2024-05-04T18:18:54.446 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:54.446 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 10 monclient: _finish_auth 0 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 10 monclient: _check_auth_tickets 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 1 -- 172.21.15.19:0/3604367410 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f2c00c940 con 0x7f9f3c0a9fb0 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 1 -- 172.21.15.19:0/3604367410 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c00cd30 con 0x7f9f3c0a9fb0 2024-05-04T18:18:54.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f32ffd640 10 monclient: dump: 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f40e0e640 5 monclient: authenticate success, global_id 18797 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f40e0e640 10 monclient: _renew_subs 2024-05-04T18:18:54.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f40e0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:54.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f40e0e640 1 -- 172.21.15.19:0/3604367410 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f3c1368a0 con 0x7f9f3c0a9fb0 2024-05-04T18:18:54.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f3959a640 4 set_mon_vals no callback set 2024-05-04T18:18:54.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f40e0e640 10 client.18797.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:54.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f40e0e640 10 monclient: _renew_subs 2024-05-04T18:18:54.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f40e0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:54.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f40e0e640 1 -- 172.21.15.19:0/3604367410 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f3c134c30 con 0x7f9f3c0a9fb0 2024-05-04T18:18:54.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.444+0000 7f9f40e0e640 1 librados: init done 2024-05-04T18:18:54.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.445+0000 7f9f40e0e640 10 client.18797.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:54.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.445+0000 7f9f40e0e640 15 client.18797.objecter _session_command_op_assign -1 1 2024-05-04T18:18:54.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.445+0000 7f9f40e0e640 20 client.18797.objecter _get_session osd=-1 returning homeless 2024-05-04T18:18:54.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.445+0000 7f9f40e0e640 10 client.18797.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:54.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.445+0000 7f9f40e0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:54.450 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.445+0000 7f9f40e0e640 1 -- 172.21.15.19:0/3604367410 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9f3c137df0 con 0x7f9f3c0a9fb0 2024-05-04T18:18:54.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.465+0000 7f9f32ffd640 1 -- 172.21.15.19:0/3604367410 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9f2c01a070 con 0x7f9f3c0a9fb0 2024-05-04T18:18:54.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.466+0000 7f9f32ffd640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f9f1c06d810 0x7f9f1c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.467 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.466+0000 7f9f33fff640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f9f1c06d810 0x7f9f1c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.466+0000 7f9f33fff640 10 monclient: get_auth_request con 0x7f9f1c06d810 auth_method 0 2024-05-04T18:18:54.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.467+0000 7f9f32ffd640 10 client.18797.objecter ms_handle_connect 0x7f9f1c06d810 2024-05-04T18:18:54.468 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.467+0000 7f9f33fff640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f9f1c06d810 0x7f9f1c06fcd0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9f24036000 tx=0x7f9f2403c040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:54.471 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 1 -- 172.21.15.19:0/3604367410 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f9f2c014de0 con 0x7f9f3c0a9fb0 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 10 client.18797.objecter ms_dispatch 0x7f9f3c000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 3 client.18797.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 3 client.18797.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 10 client.18797.objecter checking command 1 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] conn(0x7f9f1c075670 0x7f9f1c077b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 20 client.18797.objecter _get_session s=0x7f9f1c074fa0 osd=2 3 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 20 client.18797.objecter put_session s=0x7f9f1c074fa0 osd=2 3 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 15 client.18797.objecter _session_command_op_remove -1 1 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 20 client.18797.objecter _get_session s=0x7f9f1c074fa0 osd=2 3 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 20 client.18797.objecter get_session s=0x7f9f1c074fa0 osd=2 3 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 15 client.18797.objecter _session_command_op_assign 2 1 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 20 client.18797.objecter put_session s=0x7f9f1c074fa0 osd=2 4 2024-05-04T18:18:54.472 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 10 client.18797.objecter _send_command 1 2024-05-04T18:18:54.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 1 -- 172.21.15.19:0/3604367410 --> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f2c022e50 con 0x7f9f1c075670 2024-05-04T18:18:54.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 20 client.18797.objecter dump_active .. 0 homeless 2024-05-04T18:18:54.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 1 -- 172.21.15.19:0/3604367410 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_get_version_reply(handle=1 version=1186) ==== 24+0+0 (secure 0 0 0) 0x7f9f2c055640 con 0x7f9f3c0a9fb0 2024-05-04T18:18:54.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f32ffd640 10 monclient: handle_get_version_reply finishing 1 version 1186 2024-05-04T18:18:54.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.470+0000 7f9f3ad9d640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] conn(0x7f9f1c075670 0x7f9f1c077b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.473 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.471+0000 7f9f3ad9d640 10 monclient: get_auth_request con 0x7f9f1c075670 auth_method 0 2024-05-04T18:18:54.474 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.473+0000 7f9f3ad9d640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] conn(0x7f9f1c075670 0x7f9f1c077b10 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:54.475 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.473+0000 7f9f32ffd640 10 client.18797.objecter ms_handle_connect 0x7f9f1c075670 2024-05-04T18:18:54.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.499+0000 7f9f32ffd640 1 -- 172.21.15.19:0/3604367410 <== osd.2 v2:172.21.15.19:6816/3744614279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9f2c022e50 con 0x7f9f1c075670 2024-05-04T18:18:54.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.499+0000 7f9f32ffd640 10 client.18797.objecter ms_dispatch 0x7f9f3c000b90 command_reply(tid 1: 0 ) 2024-05-04T18:18:54.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.499+0000 7f9f32ffd640 10 client.18797.objecter _finish_command 1 = system:0 2024-05-04T18:18:54.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.499+0000 7f9f32ffd640 20 client.18797.objecter put_session s=0x7f9f1c074fa0 osd=2 4 2024-05-04T18:18:54.501 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.499+0000 7f9f32ffd640 15 client.18797.objecter _session_command_op_remove 2 1 2024-05-04T18:18:54.515 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.514+0000 7f9f40e0e640 10 client.18797.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-04T18:18:54.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.514+0000 7f9f40e0e640 15 client.18797.objecter _session_command_op_assign -1 2 2024-05-04T18:18:54.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.514+0000 7f9f40e0e640 20 client.18797.objecter _get_session s=0x7f9f1c074fa0 osd=2 3 2024-05-04T18:18:54.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.514+0000 7f9f40e0e640 20 client.18797.objecter put_session s=0x7f9f1c074fa0 osd=2 3 2024-05-04T18:18:54.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.514+0000 7f9f40e0e640 20 client.18797.objecter _get_session s=0x7f9f1c074fa0 osd=2 3 2024-05-04T18:18:54.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.514+0000 7f9f40e0e640 15 client.18797.objecter _session_command_op_remove -1 2 2024-05-04T18:18:54.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.514+0000 7f9f40e0e640 20 client.18797.objecter get_session s=0x7f9f1c074fa0 osd=2 3 2024-05-04T18:18:54.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.514+0000 7f9f40e0e640 15 client.18797.objecter _session_command_op_assign 2 2 2024-05-04T18:18:54.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.514+0000 7f9f40e0e640 20 client.18797.objecter put_session s=0x7f9f1c074fa0 osd=2 4 2024-05-04T18:18:54.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.514+0000 7f9f40e0e640 10 client.18797.objecter _send_command 2 2024-05-04T18:18:54.516 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.514+0000 7f9f40e0e640 1 -- 172.21.15.19:0/3604367410 --> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9f3c068b10 con 0x7f9f1c075670 2024-05-04T18:18:54.529 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.528+0000 7f9f32ffd640 1 -- 172.21.15.19:0/3604367410 <== osd.2 v2:172.21.15.19:6816/3744614279 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9f3c068b10 con 0x7f9f1c075670 2024-05-04T18:18:54.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.528+0000 7f9f32ffd640 10 client.18797.objecter ms_dispatch 0x7f9f3c000b90 command_reply(tid 2: 0 ) 2024-05-04T18:18:54.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.528+0000 7f9f32ffd640 10 client.18797.objecter _finish_command 2 = system:0 2024-05-04T18:18:54.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.528+0000 7f9f32ffd640 20 client.18797.objecter put_session s=0x7f9f1c074fa0 osd=2 4 2024-05-04T18:18:54.530 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.528+0000 7f9f32ffd640 15 client.18797.objecter _session_command_op_remove 2 2 2024-05-04T18:18:54.530 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-05-04T18:18:54.530 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T18:18:54.530 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-05-04T18:18:54.531 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.528+0000 7f9f40e0e640 10 librados: watch_flush enter 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 10 librados: watch_flush exit 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 10 client.18797.objecter close_session for osd.2 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 1 -- 172.21.15.19:0/3604367410 >> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] conn(0x7f9f1c075670 msgr2=0x7f9f1c077b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.19:6816/3744614279,v1:172.21.15.19:6817/3744614279] conn(0x7f9f1c075670 0x7f9f1c077b10 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 20 client.18797.objecter put_session s=0x7f9f1c074fa0 osd=2 1 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 20 client.18797.objecter shutdown clearing up homeless session... 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 10 client.18797.objecter successfully canceled tick 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 1 -- 172.21.15.19:0/3604367410 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f9f1c06d810 msgr2=0x7f9f1c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f9f1c06d810 0x7f9f1c06fcd0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9f24036000 tx=0x7f9f2403c040 comp rx=0 tx=0).stop 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 10 monclient: shutdown 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 1 -- 172.21.15.19:0/3604367410 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0a9fb0 msgr2=0x7f9f3c13a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 1 --2- 172.21.15.19:0/3604367410 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9f3c0a9fb0 0x7f9f3c13a4b0 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c00e8a0 tx=0x7f9f2c00ed60 comp rx=0 tx=0).stop 2024-05-04T18:18:54.532 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f3ad9d640 1 -- 172.21.15.19:0/3604367410 reap_dead start 2024-05-04T18:18:54.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 1 -- 172.21.15.19:0/3604367410 shutdown_connections 2024-05-04T18:18:54.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 1 -- 172.21.15.19:0/3604367410 >> 172.21.15.19:0/3604367410 conn(0x7f9f3c0b4e90 msgr2=0x7f9f3c0b5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:54.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 1 -- 172.21.15.19:0/3604367410 shutdown_connections 2024-05-04T18:18:54.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 1 -- 172.21.15.19:0/3604367410 wait complete. 2024-05-04T18:18:54.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.529+0000 7f9f40e0e640 1 librados: shutdown 2024-05-04T18:18:54.541 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-04T18:18:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.640+0000 7f8c4209a640 10 monclient: get_monmap_and_config 2024-05-04T18:18:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.640+0000 7f8c4209a640 10 monclient: build_initial_monmap 2024-05-04T18:18:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.640+0000 7f8c4209a640 10 monclient: monmap: 2024-05-04T18:18:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:54.642763+0000 2024-05-04T18:18:54.642 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:54.642763+0000 2024-05-04T18:18:54.643 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:54.643 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:54.643 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:54.643 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:54.643 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:54.643 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:54.643 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:54.643 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:54.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.641+0000 7f8c4209a640 1 Processor -- start 2024-05-04T18:18:54.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.641+0000 7f8c4209a640 1 -- start start 2024-05-04T18:18:54.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.641+0000 7f8c4209a640 10 monclient: init 2024-05-04T18:18:54.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:54.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-05-04T18:18:54.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c3c0afe10 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:54.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f8c3c13ca20 0x7f8c3c072880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 10 monclient(hunting): picked mon.noname-a con 0x7f8c3c13ca20 addr v2:172.21.15.19:3300/0 2024-05-04T18:18:54.645 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8c3c0b13a0 0x7f8c3c072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 10 monclient(hunting): picked mon.noname-b con 0x7f8c3c0b13a0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c3c06fd30 con 0x7f8c3c0afe10 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f8c3c0b01f0 con 0x7f8c3c13ca20 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f8c3c070c30 con 0x7f8c3c0b13a0 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.642+0000 7f8c4209a640 10 monclient(hunting): authenticate will time out at 3356.756984s 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3bfff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8c3c0b13a0 0x7f8c3c072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3affd640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f8c3c13ca20 0x7f8c3c072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3b7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8c3c0afe10 0x7f8c3c13c310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:44512/0 (socket says 172.21.15.19:44512) 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3b7fe640 1 -- 172.21.15.19:0/4080033211 learned_addr learned my addr 172.21.15.19:0/4080033211 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3bfff640 10 monclient(hunting): get_auth_request con 0x7f8c3c0b13a0 auth_method 0 2024-05-04T18:18:54.646 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3bfff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3affd640 10 monclient(hunting): get_auth_request con 0x7f8c3c13ca20 auth_method 0 2024-05-04T18:18:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3affd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.643+0000 7f8c3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.644+0000 7f8c3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.644+0000 7f8c3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:54.647 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.644+0000 7f8c3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.645+0000 7f8c3bfff640 10 monclient(hunting): handle_auth_done global_id 18800 payload 274 2024-05-04T18:18:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.645+0000 7f8c3bfff640 1 -- 172.21.15.19:0/4080033211 >> v2:172.21.15.19:3300/0 conn(0x7f8c3c13ca20 msgr2=0x7f8c3c072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.645+0000 7f8c3bfff640 1 --2- 172.21.15.19:0/4080033211 >> v2:172.21.15.19:3300/0 conn(0x7f8c3c13ca20 0x7f8c3c072880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.645+0000 7f8c3bfff640 1 -- 172.21.15.19:0/4080033211 >> v1:172.21.15.186:6789/0 conn(0x7f8c3c0afe10 legacy=0x7f8c3c13c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.645+0000 7f8c3bfff640 10 monclient: _finish_hunting 0 2024-05-04T18:18:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.645+0000 7f8c3bfff640 1 monclient: found mon.noname-b 2024-05-04T18:18:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.645+0000 7f8c3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.645+0000 7f8c3bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T18:18:54.648 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.645+0000 7f8c3bfff640 1 -- 172.21.15.19:0/4080033211 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c3c0705a0 con 0x7f8c3c0b13a0 2024-05-04T18:18:54.649 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.647+0000 7f8c3bfff640 1 --2- 172.21.15.19:0/4080033211 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8c3c0b13a0 0x7f8c3c072dc0 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009d90 tx=0x7f8c3402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60ca325f723ad7fc server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:54.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 1 -- 172.21.15.19:0/4080033211 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c34039070 con 0x7f8c3c0b13a0 2024-05-04T18:18:54.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:54.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:54.653 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 10 monclient: dump: 2024-05-04T18:18:54.653 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:54.653 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:54.653 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.653 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.653 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:54.653 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:54.653 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 10 monclient: _finish_auth 0 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 10 monclient: _check_auth_tickets 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 1 -- 172.21.15.19:0/4080033211 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c3402cbc0 con 0x7f8c3c0b13a0 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 1 -- 172.21.15.19:0/4080033211 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c34034440 con 0x7f8c3c0b13a0 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c4209a640 5 monclient: authenticate success, global_id 18800 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c4209a640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c4209a640 4 set_mon_vals no callback set 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:54.654 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.651+0000 7f8c3a7fc640 10 monclient: dump: 2024-05-04T18:18:54.655 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:54.655 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:54.655 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.655 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.655 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:54.655 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:54.655 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:54.655 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:54.655 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:54.655 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:54.658 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.652+0000 7f8c4209a640 10 monclient: shutdown 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.652+0000 7f8c4209a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.652+0000 7f8c4209a640 1 -- 172.21.15.19:0/4080033211 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8c3c0b13a0 msgr2=0x7f8c3c072dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.652+0000 7f8c4209a640 1 --2- 172.21.15.19:0/4080033211 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8c3c0b13a0 0x7f8c3c072dc0 secure :-1 s=READY pgs=3167 cs=0 l=1 rev1=1 crypto rx=0x7f8c34009d90 tx=0x7f8c3402be10 comp rx=0 tx=0).stop 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.652+0000 7f8c4209a640 1 -- 172.21.15.19:0/4080033211 shutdown_connections 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.652+0000 7f8c4209a640 1 --2- 172.21.15.19:0/4080033211 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8c3c0b13a0 0x7f8c3c072dc0 unknown :-1 s=CLOSED pgs=3167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.652+0000 7f8c4209a640 1 --2- 172.21.15.19:0/4080033211 >> v2:172.21.15.19:3300/0 conn(0x7f8c3c13ca20 0x7f8c3c072880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.652+0000 7f8c4209a640 1 -- 172.21.15.19:0/4080033211 >> 172.21.15.19:0/4080033211 conn(0x7f8c3c0b4870 msgr2=0x7f8c3c0b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.652+0000 7f8c4209a640 1 -- 172.21.15.19:0/4080033211 shutdown_connections 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.652+0000 7f8c4209a640 1 -- 172.21.15.19:0/4080033211 wait complete. 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.653+0000 7f8c4209a640 10 monclient: build_initial_monmap 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.653+0000 7f8c4209a640 10 monclient: monmap: 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:54.659 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.653+0000 7f8c4209a640 1 librados: starting msgr at 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.653+0000 7f8c4209a640 1 librados: starting objecter 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 1 Processor -- start 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 1 -- start start 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 1 librados: setting wanted keys 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 1 librados: calling monclient init 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 10 monclient: init 2024-05-04T18:18:54.660 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8c3c0afe10 0x7f8c3c134c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 10 monclient(hunting): picked mon.noname-c con 0x7f8c3c0afe10 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c3c0b13a0 0x7f8c3c1426f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 10 monclient(hunting): picked mon.noname-a con 0x7f8c3c0b13a0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3c13ca20 0x7f8c3c13d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 10 monclient(hunting): picked mon.noname-b con 0x7f8c3c13ca20 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c0b19a0 con 0x7f8c3c0b13a0 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3c0b1820 con 0x7f8c3c13ca20 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f8c3c073300 con 0x7f8c3c0afe10 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:54.661 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.654+0000 7f8c4209a640 10 monclient(hunting): authenticate will time out at 3356.769473s 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3b7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8c3c0afe10 0x7f8c3c134c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3affd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c3c0b13a0 0x7f8c3c1426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3b7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8c3c0afe10 0x7f8c3c134c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58146/0 (socket says 172.21.15.19:58146) 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3b7fe640 1 -- 172.21.15.19:0/2528652478 learned_addr learned my addr 172.21.15.19:0/2528652478 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3b7fe640 10 monclient(hunting): get_auth_request con 0x7f8c3c0afe10 auth_method 0 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3bfff640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3c13ca20 0x7f8c3c13d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3affd640 10 monclient(hunting): get_auth_request con 0x7f8c3c0b13a0 auth_method 0 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3affd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.655+0000 7f8c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3bfff640 10 monclient(hunting): get_auth_request con 0x7f8c3c13ca20 auth_method 0 2024-05-04T18:18:54.662 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:54.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3bfff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:54.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:54.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:54.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:54.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:54.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:54.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:54.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:54.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:54.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:54.663 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.656+0000 7f8c3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.659+0000 7f8c3bfff640 10 monclient(hunting): handle_auth_done global_id 22756 payload 824 2024-05-04T18:18:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.659+0000 7f8c3bfff640 1 -- 172.21.15.19:0/2528652478 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8c3c0afe10 msgr2=0x7f8c3c134c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.659+0000 7f8c3bfff640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8c3c0afe10 0x7f8c3c134c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.659+0000 7f8c3bfff640 1 -- 172.21.15.19:0/2528652478 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c3c0b13a0 msgr2=0x7f8c3c1426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.659+0000 7f8c3bfff640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c3c0b13a0 0x7f8c3c1426f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.659+0000 7f8c3bfff640 10 monclient: _finish_hunting 0 2024-05-04T18:18:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.659+0000 7f8c3bfff640 1 monclient: found mon.noname-b 2024-05-04T18:18:54.664 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.659+0000 7f8c3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.659+0000 7f8c3bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.659+0000 7f8c3bfff640 1 -- 172.21.15.19:0/2528652478 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c3c1432f0 con 0x7f8c3c13ca20 2024-05-04T18:18:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.661+0000 7f8c3bfff640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3c13ca20 0x7f8c3c13d9c0 secure :-1 s=READY pgs=5828 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c070db0 tx=0x7f8c34004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.661+0000 7f8c23fff640 10 client.?.objecter ms_handle_connect 0x7f8c3c13ca20 2024-05-04T18:18:54.665 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.661+0000 7f8c23fff640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 1 -- 172.21.15.19:0/2528652478 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c34044020 con 0x7f8c3c13ca20 2024-05-04T18:18:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 10 monclient: dump: 2024-05-04T18:18:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:54.667 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 10 monclient: _finish_auth 0 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 10 monclient: _check_auth_tickets 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 1 -- 172.21.15.19:0/2528652478 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c3403f470 con 0x7f8c3c13ca20 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 1 -- 172.21.15.19:0/2528652478 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c34039040 con 0x7f8c3c13ca20 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c4209a640 5 monclient: authenticate success, global_id 22756 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c4209a640 10 monclient: _renew_subs 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c4209a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c4209a640 1 -- 172.21.15.19:0/2528652478 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c3c13d5a0 con 0x7f8c3c13ca20 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c397fa640 4 set_mon_vals no callback set 2024-05-04T18:18:54.668 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c23fff640 10 monclient: dump: 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c4209a640 10 client.22756.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c4209a640 10 monclient: _renew_subs 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c4209a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:54.669 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c4209a640 1 -- 172.21.15.19:0/2528652478 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c3c141530 con 0x7f8c3c13ca20 2024-05-04T18:18:54.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.665+0000 7f8c4209a640 1 librados: init done 2024-05-04T18:18:54.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.666+0000 7f8c4209a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:54.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.666+0000 7f8c4209a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:54.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.666+0000 7f8c4209a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:54.670 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.666+0000 7f8c4209a640 1 -- 172.21.15.19:0/2528652478 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c3c141fd0 con 0x7f8c3c13ca20 2024-05-04T18:18:54.686 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.685+0000 7f8c23fff640 1 -- 172.21.15.19:0/2528652478 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8c34047020 con 0x7f8c3c13ca20 2024-05-04T18:18:54.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.685+0000 7f8c23fff640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f8c2806d810 0x7f8c2806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:54.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.686+0000 7f8c3b7fe640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f8c2806d810 0x7f8c2806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:54.687 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.686+0000 7f8c3b7fe640 10 monclient: get_auth_request con 0x7f8c2806d810 auth_method 0 2024-05-04T18:18:54.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.686+0000 7f8c3b7fe640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f8c2806d810 0x7f8c2806fcd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c004e40 tx=0x7f8c2c007f10 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:54.688 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.686+0000 7f8c23fff640 10 client.22756.objecter ms_handle_connect 0x7f8c2806d810 2024-05-04T18:18:54.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.921+0000 7f8c23fff640 1 -- 172.21.15.19:0/2528652478 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f8c34079f60 con 0x7f8c3c13ca20 2024-05-04T18:18:54.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.921+0000 7f8c23fff640 10 client.22756.objecter ms_dispatch 0x7f8c3c000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:54.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.921+0000 7f8c23fff640 3 client.22756.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:54.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.921+0000 7f8c23fff640 3 client.22756.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:54.923 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.921+0000 7f8c23fff640 20 client.22756.objecter dump_active .. 0 homeless 2024-05-04T18:18:54.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.930+0000 7f8c23fff640 1 -- 172.21.15.19:0/2528652478 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8c3404e030 con 0x7f8c3c13ca20 2024-05-04T18:18:54.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.930+0000 7f8c23fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:54.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:54.930+0000 7f8c23fff640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:18:55.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.124+0000 7f8c4209a640 1 -- 172.21.15.19:0/2528652478 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f8c3c141760 con 0x7f8c2806d810 2024-05-04T18:18:55.126 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.125+0000 7f8c23fff640 1 -- 172.21.15.19:0/2528652478 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8c3c141760 con 0x7f8c2806d810 2024-05-04T18:18:55.126 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 2 to deep-scrub 2024-05-04T18:18:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 10 librados: watch_flush enter 2024-05-04T18:18:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 10 librados: watch_flush exit 2024-05-04T18:18:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 20 client.22756.objecter shutdown clearing up homeless session... 2024-05-04T18:18:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 10 client.22756.objecter successfully canceled tick 2024-05-04T18:18:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 -- 172.21.15.19:0/2528652478 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f8c2806d810 msgr2=0x7f8c2806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f8c2806d810 0x7f8c2806fcd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c004e40 tx=0x7f8c2c007f10 comp rx=0 tx=0).stop 2024-05-04T18:18:55.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 10 monclient: shutdown 2024-05-04T18:18:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 -- 172.21.15.19:0/2528652478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3c13ca20 msgr2=0x7f8c3c13d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3c13ca20 0x7f8c3c13d9c0 secure :-1 s=READY pgs=5828 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c070db0 tx=0x7f8c34004a40 comp rx=0 tx=0).stop 2024-05-04T18:18:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 -- 172.21.15.19:0/2528652478 shutdown_connections 2024-05-04T18:18:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f8c2806d810 0x7f8c2806fcd0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c3c13ca20 0x7f8c3c13d9c0 unknown :-1 s=CLOSED pgs=5828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f8c3c0b13a0 0x7f8c3c1426f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 --2- 172.21.15.19:0/2528652478 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f8c3c0afe10 0x7f8c3c134c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 -- 172.21.15.19:0/2528652478 >> 172.21.15.19:0/2528652478 conn(0x7f8c3c0b4870 msgr2=0x7f8c3c06f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 -- 172.21.15.19:0/2528652478 shutdown_connections 2024-05-04T18:18:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 -- 172.21.15.19:0/2528652478 wait complete. 2024-05-04T18:18:55.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.128+0000 7f8c4209a640 1 librados: shutdown 2024-05-04T18:18:55.145 INFO:tasks.ceph:Scrubbing osd.3 2024-05-04T18:18:55.145 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-04T18:18:55.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.257+0000 7fae59bbd640 10 monclient: get_monmap_and_config 2024-05-04T18:18:55.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.257+0000 7fae59bbd640 10 monclient: build_initial_monmap 2024-05-04T18:18:55.259 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.257+0000 7fae59bbd640 10 monclient: monmap: 2024-05-04T18:18:55.259 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:55.259 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:55.259 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:55.259620+0000 2024-05-04T18:18:55.259 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:55.259620+0000 2024-05-04T18:18:55.259 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:55.259 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:55.260 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:55.260 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:55.260 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:55.260 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:55.260 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:55.260 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:55.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.258+0000 7fae59bbd640 1 Processor -- start 2024-05-04T18:18:55.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.258+0000 7fae59bbd640 1 -- start start 2024-05-04T18:18:55.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.258+0000 7fae59bbd640 10 monclient: init 2024-05-04T18:18:55.261 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.259+0000 7fae59bbd640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:55.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.259+0000 7fae59bbd640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-05-04T18:18:55.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.259+0000 7fae59bbd640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fae540a9fb0 0x7fae540aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:55.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.259+0000 7fae59bbd640 10 monclient(hunting): picked mon.noname-a con 0x7fae540a9fb0 addr v2:172.21.15.19:3300/0 2024-05-04T18:18:55.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.259+0000 7fae59bbd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae540b27f0 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:55.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.259+0000 7fae59bbd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae541404e0 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:55.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.259+0000 7fae59bbd640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae540ab880 con 0x7fae541404e0 2024-05-04T18:18:55.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.259+0000 7fae59bbd640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae54146b00 con 0x7fae540b27f0 2024-05-04T18:18:55.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.259+0000 7fae59bbd640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:55.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.259+0000 7fae59bbd640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fae54146cd0 con 0x7fae540a9fb0 2024-05-04T18:18:55.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.259+0000 7fae59bbd640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:55.262 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.259+0000 7fae59bbd640 10 monclient(hunting): authenticate will time out at 3357.374544s 2024-05-04T18:18:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.299+0000 7fae52ffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fae540b27f0 0x7fae5413fdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:44522/0 (socket says 172.21.15.19:44522) 2024-05-04T18:18:55.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.299+0000 7fae52ffd640 1 -- 172.21.15.19:0/2803609168 learned_addr learned my addr 172.21.15.19:0/2803609168 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T18:18:55.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.316+0000 7fae537fe640 1 --2- 172.21.15.19:0/2803609168 >> v2:172.21.15.19:3300/0 conn(0x7fae540a9fb0 0x7fae540aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:55.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.316+0000 7fae537fe640 10 monclient(hunting): get_auth_request con 0x7fae540a9fb0 auth_method 0 2024-05-04T18:18:55.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.317+0000 7fae537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:55.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.317+0000 7fae537fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:55.318 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.317+0000 7fae537fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:55.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.318+0000 7fae537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:55.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.318+0000 7fae537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:55.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.318+0000 7fae537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:55.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.318+0000 7fae527fc640 1 -- 172.21.15.19:0/2803609168 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443461752 0 0) 0x7fae54146b00 con 0x7fae540b27f0 2024-05-04T18:18:55.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.318+0000 7fae527fc640 10 monclient(hunting): _negotiate 2024-05-04T18:18:55.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.318+0000 7fae527fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:55.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.318+0000 7fae527fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:55.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.318+0000 7fae527fc640 10 monclient(hunting): my global_id is 22759 2024-05-04T18:18:55.320 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.318+0000 7fae527fc640 1 -- 172.21.15.19:0/2803609168 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae3c003520 con 0x7fae540b27f0 2024-05-04T18:18:55.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.322+0000 7fae537fe640 10 monclient(hunting): handle_auth_done global_id 15795 payload 274 2024-05-04T18:18:55.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.322+0000 7fae537fe640 1 -- 172.21.15.19:0/2803609168 >> v1:172.21.15.186:6789/0 conn(0x7fae540b27f0 legacy=0x7fae5413fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.322+0000 7fae537fe640 1 -- 172.21.15.19:0/2803609168 >> v1:172.21.15.19:6789/0 conn(0x7fae541404e0 legacy=0x7fae541463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.322+0000 7fae537fe640 10 monclient: _finish_hunting 0 2024-05-04T18:18:55.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.322+0000 7fae537fe640 1 monclient: found mon.noname-a 2024-05-04T18:18:55.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.322+0000 7fae537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:55.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.322+0000 7fae537fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T18:18:55.324 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.322+0000 7fae537fe640 1 -- 172.21.15.19:0/2803609168 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae54146e50 con 0x7fae540a9fb0 2024-05-04T18:18:55.326 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.325+0000 7fae537fe640 1 --2- 172.21.15.19:0/2803609168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae540a9fb0 0x7fae540aa390 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7fae44009d90 tx=0x7fae4402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=787e5243789f1c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:55.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 1 -- 172.21.15.19:0/2803609168 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae44039070 con 0x7fae540a9fb0 2024-05-04T18:18:55.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:55.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T18:18:55.337 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 10 monclient: dump: 2024-05-04T18:18:55.337 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:55.337 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:55.337 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.337 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.337 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:55.337 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:55.337 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:55.337 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 10 monclient: _finish_auth 0 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 10 monclient: _check_auth_tickets 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 1 -- 172.21.15.19:0/2803609168 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae4402cc60 con 0x7fae540a9fb0 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae59bbd640 5 monclient: authenticate success, global_id 15795 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae59bbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae59bbd640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae59bbd640 4 set_mon_vals no callback set 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 1 -- 172.21.15.19:0/2803609168 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae44034660 con 0x7fae540a9fb0 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:55.338 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T18:18:55.339 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.335+0000 7fae527fc640 10 monclient: dump: 2024-05-04T18:18:55.339 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:55.339 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:55.339 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.339 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.339 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:55.339 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:55.339 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:55.339 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:55.339 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:55.339 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:55.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.336+0000 7fae59bbd640 10 monclient: shutdown 2024-05-04T18:18:55.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.336+0000 7fae59bbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:55.343 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.336+0000 7fae59bbd640 1 -- 172.21.15.19:0/2803609168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae540a9fb0 msgr2=0x7fae540aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.336+0000 7fae59bbd640 1 --2- 172.21.15.19:0/2803609168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae540a9fb0 0x7fae540aa390 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7fae44009d90 tx=0x7fae4402beb0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.336+0000 7fae59bbd640 1 -- 172.21.15.19:0/2803609168 shutdown_connections 2024-05-04T18:18:55.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.336+0000 7fae59bbd640 1 --2- 172.21.15.19:0/2803609168 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae540a9fb0 0x7fae540aa390 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.336+0000 7fae59bbd640 1 -- 172.21.15.19:0/2803609168 >> 172.21.15.19:0/2803609168 conn(0x7fae540b4e90 msgr2=0x7fae540b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.336+0000 7fae59bbd640 1 -- 172.21.15.19:0/2803609168 shutdown_connections 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.336+0000 7fae59bbd640 1 -- 172.21.15.19:0/2803609168 wait complete. 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.337+0000 7fae59bbd640 10 monclient: build_initial_monmap 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.337+0000 7fae59bbd640 10 monclient: monmap: 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:55.345 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.337+0000 7fae59bbd640 1 librados: starting msgr at 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.337+0000 7fae59bbd640 1 librados: starting objecter 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.337+0000 7fae59bbd640 1 Processor -- start 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.337+0000 7fae59bbd640 1 -- start start 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.337+0000 7fae59bbd640 1 librados: setting wanted keys 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.337+0000 7fae59bbd640 1 librados: calling monclient init 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.337+0000 7fae59bbd640 10 monclient: init 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 10 monclient: _add_conns ranks=[2,0,1] 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fae540a9fb0 0x7fae541328d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 10 monclient(hunting): picked mon.noname-c con 0x7fae540a9fb0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae540b27f0 0x7fae54133320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 10 monclient(hunting): picked mon.noname-a con 0x7fae540b27f0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:55.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae541404e0 0x7fae541db920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 10 monclient(hunting): picked mon.noname-b con 0x7fae541404e0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fae5414e7b0 con 0x7fae540b27f0 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fae5414e630 con 0x7fae541404e0 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fae5414e930 con 0x7fae540a9fb0 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.338+0000 7fae59bbd640 10 monclient(hunting): authenticate will time out at 3357.453096s 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.339+0000 7fae53fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae541404e0 0x7fae541db920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.339+0000 7fae53fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae541404e0 0x7fae541db920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:45376/0 (socket says 172.21.15.19:45376) 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.339+0000 7fae53fff640 1 -- 172.21.15.19:0/251501541 learned_addr learned my addr 172.21.15.19:0/251501541 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.339+0000 7fae53fff640 10 monclient(hunting): get_auth_request con 0x7fae541404e0 auth_method 0 2024-05-04T18:18:55.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.339+0000 7fae53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.339+0000 7fae53fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.339+0000 7fae53fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.339+0000 7fae52ffd640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae540b27f0 0x7fae54133320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.340+0000 7fae53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.340+0000 7fae53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.340+0000 7fae53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.340+0000 7fae52ffd640 10 monclient(hunting): get_auth_request con 0x7fae540b27f0 auth_method 0 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.340+0000 7fae52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.340+0000 7fae52ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.340+0000 7fae52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.340+0000 7fae52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.340+0000 7fae52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.340+0000 7fae52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.340+0000 7fae537fe640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fae540a9fb0 0x7fae541328d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.341+0000 7fae537fe640 10 monclient(hunting): get_auth_request con 0x7fae540a9fb0 auth_method 0 2024-05-04T18:18:55.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.341+0000 7fae537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:55.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.341+0000 7fae537fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:55.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.341+0000 7fae537fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:55.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.341+0000 7fae537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:55.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.341+0000 7fae537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:55.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.341+0000 7fae537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:55.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.343+0000 7fae53fff640 10 monclient(hunting): handle_auth_done global_id 22762 payload 824 2024-05-04T18:18:55.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.343+0000 7fae53fff640 1 -- 172.21.15.19:0/251501541 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fae540a9fb0 msgr2=0x7fae541328d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.343+0000 7fae53fff640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fae540a9fb0 0x7fae541328d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.343+0000 7fae53fff640 1 -- 172.21.15.19:0/251501541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae540b27f0 msgr2=0x7fae54133320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.343+0000 7fae53fff640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fae540b27f0 0x7fae54133320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.343+0000 7fae53fff640 10 monclient: _finish_hunting 0 2024-05-04T18:18:55.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.343+0000 7fae53fff640 1 monclient: found mon.noname-b 2024-05-04T18:18:55.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.343+0000 7fae53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:55.351 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.343+0000 7fae53fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.343+0000 7fae53fff640 1 -- 172.21.15.19:0/251501541 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae54134f80 con 0x7fae541404e0 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.345+0000 7fae53fff640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae541404e0 0x7fae541db920 secure :-1 s=READY pgs=5830 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00c3a0 tx=0x7fae4c00c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.345+0000 7fae3bfff640 10 client.?.objecter ms_handle_connect 0x7fae541404e0 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.345+0000 7fae3bfff640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 1 -- 172.21.15.19:0/251501541 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4c017070 con 0x7fae541404e0 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 10 monclient: dump: 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:55.352 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 10 monclient: _finish_auth 0 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 10 monclient: _check_auth_tickets 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 1 -- 172.21.15.19:0/251501541 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae4c0052d0 con 0x7fae541404e0 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae59bbd640 5 monclient: authenticate success, global_id 22762 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae59bbd640 10 monclient: _renew_subs 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 1 -- 172.21.15.19:0/251501541 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4c003830 con 0x7fae541404e0 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae59bbd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae59bbd640 1 -- 172.21.15.19:0/251501541 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae54134ca0 con 0x7fae541404e0 2024-05-04T18:18:55.353 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae3bfff640 10 monclient: dump: 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:55.354 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae51ffb640 4 set_mon_vals no callback set 2024-05-04T18:18:55.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae59bbd640 10 client.22762.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:55.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae59bbd640 10 monclient: _renew_subs 2024-05-04T18:18:55.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae59bbd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:55.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae59bbd640 1 -- 172.21.15.19:0/251501541 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae54133ef0 con 0x7fae541404e0 2024-05-04T18:18:55.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.349+0000 7fae59bbd640 1 librados: init done 2024-05-04T18:18:55.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.350+0000 7fae59bbd640 10 client.22762.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:55.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.350+0000 7fae59bbd640 15 client.22762.objecter _session_command_op_assign -1 1 2024-05-04T18:18:55.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.350+0000 7fae59bbd640 20 client.22762.objecter _get_session osd=-1 returning homeless 2024-05-04T18:18:55.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.350+0000 7fae59bbd640 10 client.22762.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:55.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.350+0000 7fae59bbd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:55.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.350+0000 7fae59bbd640 1 -- 172.21.15.19:0/251501541 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fae540b75f0 con 0x7fae541404e0 2024-05-04T18:18:55.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.370+0000 7fae3bfff640 1 -- 172.21.15.19:0/251501541 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fae4c01a070 con 0x7fae541404e0 2024-05-04T18:18:55.371 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.370+0000 7fae3bfff640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fae4006d7c0 0x7fae4006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:55.372 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.370+0000 7fae537fe640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fae4006d7c0 0x7fae4006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:55.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.371+0000 7fae537fe640 10 monclient: get_auth_request con 0x7fae4006d7c0 auth_method 0 2024-05-04T18:18:55.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.371+0000 7fae3bfff640 10 client.22762.objecter ms_handle_connect 0x7fae4006d7c0 2024-05-04T18:18:55.373 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.371+0000 7fae537fe640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fae4006d7c0 0x7fae4006fc80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fae44036000 tx=0x7fae44030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:55.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 1 -- 172.21.15.19:0/251501541 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7fae4c056a10 con 0x7fae541404e0 2024-05-04T18:18:55.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 10 client.22762.objecter ms_dispatch 0x7fae54000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:55.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 3 client.22762.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:55.377 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 3 client.22762.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:55.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 10 client.22762.objecter checking command 1 2024-05-04T18:18:55.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] conn(0x7fae40075680 0x7fae40077b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:55.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 20 client.22762.objecter _get_session s=0x7fae40074fb0 osd=3 3 2024-05-04T18:18:55.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 20 client.22762.objecter put_session s=0x7fae40074fb0 osd=3 3 2024-05-04T18:18:55.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 15 client.22762.objecter _session_command_op_remove -1 1 2024-05-04T18:18:55.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 20 client.22762.objecter _get_session s=0x7fae40074fb0 osd=3 3 2024-05-04T18:18:55.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 20 client.22762.objecter get_session s=0x7fae40074fb0 osd=3 3 2024-05-04T18:18:55.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 15 client.22762.objecter _session_command_op_assign 3 1 2024-05-04T18:18:55.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 20 client.22762.objecter put_session s=0x7fae40074fb0 osd=3 4 2024-05-04T18:18:55.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 10 client.22762.objecter _send_command 1 2024-05-04T18:18:55.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 1 -- 172.21.15.19:0/251501541 --> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae4c01feb0 con 0x7fae40075680 2024-05-04T18:18:55.378 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 20 client.22762.objecter dump_active .. 0 homeless 2024-05-04T18:18:55.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 1 -- 172.21.15.19:0/251501541 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=1186) ==== 24+0+0 (secure 0 0 0) 0x7fae4c054c10 con 0x7fae541404e0 2024-05-04T18:18:55.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.375+0000 7fae3bfff640 10 monclient: handle_get_version_reply finishing 1 version 1186 2024-05-04T18:18:55.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.376+0000 7fae52ffd640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] conn(0x7fae40075680 0x7fae40077b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:55.379 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.376+0000 7fae52ffd640 10 monclient: get_auth_request con 0x7fae40075680 auth_method 0 2024-05-04T18:18:55.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.379+0000 7fae52ffd640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] conn(0x7fae40075680 0x7fae40077b20 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:55.380 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.379+0000 7fae3bfff640 10 client.22762.objecter ms_handle_connect 0x7fae40075680 2024-05-04T18:18:55.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.403+0000 7fae3bfff640 1 -- 172.21.15.19:0/251501541 <== osd.3 v2:172.21.15.19:6819/2537429684 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae4c01feb0 con 0x7fae40075680 2024-05-04T18:18:55.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.403+0000 7fae3bfff640 10 client.22762.objecter ms_dispatch 0x7fae54000b90 command_reply(tid 1: 0 ) 2024-05-04T18:18:55.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.404+0000 7fae3bfff640 10 client.22762.objecter _finish_command 1 = system:0 2024-05-04T18:18:55.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.404+0000 7fae3bfff640 20 client.22762.objecter put_session s=0x7fae40074fb0 osd=3 4 2024-05-04T18:18:55.405 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.404+0000 7fae3bfff640 15 client.22762.objecter _session_command_op_remove 3 1 2024-05-04T18:18:55.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.417+0000 7fae59bbd640 10 client.22762.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-04T18:18:55.418 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.417+0000 7fae59bbd640 15 client.22762.objecter _session_command_op_assign -1 2 2024-05-04T18:18:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.417+0000 7fae59bbd640 20 client.22762.objecter _get_session s=0x7fae40074fb0 osd=3 3 2024-05-04T18:18:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.417+0000 7fae59bbd640 20 client.22762.objecter put_session s=0x7fae40074fb0 osd=3 3 2024-05-04T18:18:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.417+0000 7fae59bbd640 20 client.22762.objecter _get_session s=0x7fae40074fb0 osd=3 3 2024-05-04T18:18:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.417+0000 7fae59bbd640 15 client.22762.objecter _session_command_op_remove -1 2 2024-05-04T18:18:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.417+0000 7fae59bbd640 20 client.22762.objecter get_session s=0x7fae40074fb0 osd=3 3 2024-05-04T18:18:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.417+0000 7fae59bbd640 15 client.22762.objecter _session_command_op_assign 3 2 2024-05-04T18:18:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.417+0000 7fae59bbd640 20 client.22762.objecter put_session s=0x7fae40074fb0 osd=3 4 2024-05-04T18:18:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.417+0000 7fae59bbd640 10 client.22762.objecter _send_command 2 2024-05-04T18:18:55.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.417+0000 7fae59bbd640 1 -- 172.21.15.19:0/251501541 --> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fae54143e80 con 0x7fae40075680 2024-05-04T18:18:55.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.430+0000 7fae3bfff640 1 -- 172.21.15.19:0/251501541 <== osd.3 v2:172.21.15.19:6819/2537429684 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fae54143e80 con 0x7fae40075680 2024-05-04T18:18:55.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.430+0000 7fae3bfff640 10 client.22762.objecter ms_dispatch 0x7fae54000b90 command_reply(tid 2: 0 ) 2024-05-04T18:18:55.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.430+0000 7fae3bfff640 10 client.22762.objecter _finish_command 2 = system:0 2024-05-04T18:18:55.432 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.430+0000 7fae3bfff640 20 client.22762.objecter put_session s=0x7fae40074fb0 osd=3 4 2024-05-04T18:18:55.433 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.430+0000 7fae3bfff640 15 client.22762.objecter _session_command_op_remove 3 2 2024-05-04T18:18:55.433 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-05-04T18:18:55.433 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T18:18:55.433 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 10 librados: watch_flush enter 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 10 librados: watch_flush exit 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 10 client.22762.objecter close_session for osd.3 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 1 -- 172.21.15.19:0/251501541 >> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] conn(0x7fae40075680 msgr2=0x7fae40077b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.19:6819/2537429684,v1:172.21.15.19:6821/2537429684] conn(0x7fae40075680 0x7fae40077b20 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 20 client.22762.objecter put_session s=0x7fae40074fb0 osd=3 1 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 20 client.22762.objecter shutdown clearing up homeless session... 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 10 client.22762.objecter successfully canceled tick 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 1 -- 172.21.15.19:0/251501541 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fae4006d7c0 msgr2=0x7fae4006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fae4006d7c0 0x7fae4006fc80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fae44036000 tx=0x7fae44030000 comp rx=0 tx=0).stop 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 10 monclient: shutdown 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:55.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 1 -- 172.21.15.19:0/251501541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae541404e0 msgr2=0x7fae541db920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 1 --2- 172.21.15.19:0/251501541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae541404e0 0x7fae541db920 secure :-1 s=READY pgs=5830 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00c3a0 tx=0x7fae4c00c860 comp rx=0 tx=0).stop 2024-05-04T18:18:55.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae53fff640 1 -- 172.21.15.19:0/251501541 reap_dead start 2024-05-04T18:18:55.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 1 -- 172.21.15.19:0/251501541 shutdown_connections 2024-05-04T18:18:55.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 1 -- 172.21.15.19:0/251501541 >> 172.21.15.19:0/251501541 conn(0x7fae540b4e90 msgr2=0x7fae540b5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:55.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 1 -- 172.21.15.19:0/251501541 shutdown_connections 2024-05-04T18:18:55.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 1 -- 172.21.15.19:0/251501541 wait complete. 2024-05-04T18:18:55.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.431+0000 7fae59bbd640 1 librados: shutdown 2024-05-04T18:18:55.444 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-04T18:18:55.547 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.546+0000 7f1d43e6a640 10 monclient: get_monmap_and_config 2024-05-04T18:18:55.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.546+0000 7f1d43e6a640 10 monclient: build_initial_monmap 2024-05-04T18:18:55.548 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.546+0000 7f1d43e6a640 10 monclient: monmap: 2024-05-04T18:18:55.548 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:55.548 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:55.548 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:55.548317+0000 2024-05-04T18:18:55.548 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:55.548317+0000 2024-05-04T18:18:55.548 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:55.548 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:55.549 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:55.549 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:55.549 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:55.549 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:55.549 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:55.550 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:55.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 1 Processor -- start 2024-05-04T18:18:55.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 1 -- start start 2024-05-04T18:18:55.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 10 monclient: init 2024-05-04T18:18:55.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:55.553 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c0a9fb0 0x7f1d3c0aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 10 monclient(hunting): picked mon.noname-b con 0x7f1d3c0a9fb0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1d3c0b27f0 0x7f1d3c13fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 10 monclient(hunting): picked mon.noname-c con 0x7f1d3c0b27f0 addr v2:172.21.15.186:3300/0 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d3c140310 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d3c0ab880 con 0x7f1d3c140310 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f1d3c1407a0 con 0x7f1d3c0b27f0 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f1d3c146b00 con 0x7f1d3c0a9fb0 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.547+0000 7f1d43e6a640 10 monclient(hunting): authenticate will time out at 3357.662597s 2024-05-04T18:18:55.554 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.548+0000 7f1d41c06640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c0a9fb0 0x7f1d3c0aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.548+0000 7f1d41405640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1d3c0b27f0 0x7f1d3c13fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.548+0000 7f1d41c06640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c0a9fb0 0x7f1d3c0aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58152/0 (socket says 172.21.15.19:58152) 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.548+0000 7f1d41c06640 1 -- 172.21.15.19:0/1673993468 learned_addr learned my addr 172.21.15.19:0/1673993468 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.548+0000 7f1d41c06640 10 monclient(hunting): get_auth_request con 0x7f1d3c0a9fb0 auth_method 0 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.548+0000 7f1d41c06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.548+0000 7f1d41c06640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.548+0000 7f1d41c06640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.548+0000 7f1d41405640 10 monclient(hunting): get_auth_request con 0x7f1d3c0b27f0 auth_method 0 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.548+0000 7f1d41405640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.548+0000 7f1d41405640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.548+0000 7f1d41405640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.549+0000 7f1d41c06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.549+0000 7f1d41c06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.549+0000 7f1d41c06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.549+0000 7f1d41405640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:55.555 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.549+0000 7f1d41405640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:55.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.549+0000 7f1d41405640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:55.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.551+0000 7f1d41c06640 10 monclient(hunting): handle_auth_done global_id 18809 payload 274 2024-05-04T18:18:55.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.551+0000 7f1d41c06640 1 -- 172.21.15.19:0/1673993468 >> v2:172.21.15.186:3300/0 conn(0x7f1d3c0b27f0 msgr2=0x7f1d3c13fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:55.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.551+0000 7f1d41c06640 1 --2- 172.21.15.19:0/1673993468 >> v2:172.21.15.186:3300/0 conn(0x7f1d3c0b27f0 0x7f1d3c13fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.551+0000 7f1d41c06640 1 -- 172.21.15.19:0/1673993468 >> v1:172.21.15.186:6789/0 conn(0x7f1d3c140310 legacy=0x7f1d3c1463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:55.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.551+0000 7f1d41c06640 10 monclient: _finish_hunting 0 2024-05-04T18:18:55.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.551+0000 7f1d41c06640 1 monclient: found mon.noname-b 2024-05-04T18:18:55.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.551+0000 7f1d41c06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:55.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.551+0000 7f1d41c06640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T18:18:55.556 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.551+0000 7f1d41c06640 1 -- 172.21.15.19:0/1673993468 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3c146cb0 con 0x7f1d3c0a9fb0 2024-05-04T18:18:55.557 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.553+0000 7f1d41c06640 1 --2- 172.21.15.19:0/1673993468 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c0a9fb0 0x7f1d3c0aa390 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009870 tx=0x7f1d3802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9134093d19c7431e server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:55.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 1 -- 172.21.15.19:0/1673993468 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d380317c0 con 0x7f1d3c0a9fb0 2024-05-04T18:18:55.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:55.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:55.562 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 10 monclient: dump: 2024-05-04T18:18:55.562 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:55.562 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 10 monclient: _finish_auth 0 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 10 monclient: _check_auth_tickets 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 1 -- 172.21.15.19:0/1673993468 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d38037070 con 0x7f1d3c0a9fb0 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 1 -- 172.21.15.19:0/1673993468 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d38031b80 con 0x7f1d3c0a9fb0 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:55.563 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d40c04640 10 monclient: dump: 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d43e6a640 5 monclient: authenticate success, global_id 18809 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d43e6a640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.556+0000 7f1d43e6a640 4 set_mon_vals no callback set 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.557+0000 7f1d43e6a640 10 monclient: shutdown 2024-05-04T18:18:55.564 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.557+0000 7f1d43e6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.557+0000 7f1d43e6a640 1 -- 172.21.15.19:0/1673993468 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c0a9fb0 msgr2=0x7f1d3c0aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.557+0000 7f1d43e6a640 1 --2- 172.21.15.19:0/1673993468 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c0a9fb0 0x7f1d3c0aa390 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009870 tx=0x7f1d3802b860 comp rx=0 tx=0).stop 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.557+0000 7f1d43e6a640 1 -- 172.21.15.19:0/1673993468 shutdown_connections 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.557+0000 7f1d43e6a640 1 --2- 172.21.15.19:0/1673993468 >> v2:172.21.15.186:3300/0 conn(0x7f1d3c0b27f0 0x7f1d3c13fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.557+0000 7f1d43e6a640 1 --2- 172.21.15.19:0/1673993468 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c0a9fb0 0x7f1d3c0aa390 unknown :-1 s=CLOSED pgs=3168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.557+0000 7f1d43e6a640 1 -- 172.21.15.19:0/1673993468 >> 172.21.15.19:0/1673993468 conn(0x7f1d3c0b4e90 msgr2=0x7f1d3c0b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.557+0000 7f1d43e6a640 1 -- 172.21.15.19:0/1673993468 shutdown_connections 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.557+0000 7f1d43e6a640 1 -- 172.21.15.19:0/1673993468 wait complete. 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.558+0000 7f1d43e6a640 10 monclient: build_initial_monmap 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.558+0000 7f1d43e6a640 10 monclient: monmap: 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:55.565 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.558+0000 7f1d43e6a640 1 librados: starting msgr at 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.558+0000 7f1d43e6a640 1 librados: starting objecter 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.558+0000 7f1d43e6a640 1 Processor -- start 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.558+0000 7f1d43e6a640 1 -- start start 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.558+0000 7f1d43e6a640 1 librados: setting wanted keys 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.558+0000 7f1d43e6a640 1 librados: calling monclient init 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.558+0000 7f1d43e6a640 10 monclient: init 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c0a9fb0 0x7f1d3c141a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:55.566 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 10 monclient(hunting): picked mon.noname-a con 0x7f1d3c0a9fb0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d3c0b27f0 0x7f1d3c141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 10 monclient(hunting): picked mon.noname-b con 0x7f1d3c0b27f0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c140310 0x7f1d3c0b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 10 monclient(hunting): picked mon.noname-c con 0x7f1d3c140310 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c14d7a0 con 0x7f1d3c0a9fb0 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c14d620 con 0x7f1d3c0b27f0 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f1d3c14d920 con 0x7f1d3c140310 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d43e6a640 10 monclient(hunting): authenticate will time out at 3357.674047s 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.559+0000 7f1d42407640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c140310 0x7f1d3c0b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41c06640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c0a9fb0 0x7f1d3c141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:55.567 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41405640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d3c0b27f0 0x7f1d3c141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d42407640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c140310 0x7f1d3c0b76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58158/0 (socket says 172.21.15.19:58158) 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d42407640 1 -- 172.21.15.19:0/444290993 learned_addr learned my addr 172.21.15.19:0/444290993 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d42407640 10 monclient(hunting): get_auth_request con 0x7f1d3c140310 auth_method 0 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d42407640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d42407640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d42407640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41405640 10 monclient(hunting): get_auth_request con 0x7f1d3c0b27f0 auth_method 0 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41405640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41405640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41405640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41c06640 10 monclient(hunting): get_auth_request con 0x7f1d3c0a9fb0 auth_method 0 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41c06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41c06640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41c06640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:55.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41405640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:55.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41405640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:55.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d41405640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:55.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d42407640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:55.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d42407640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:55.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.560+0000 7f1d42407640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:55.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.561+0000 7f1d41c06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:55.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.561+0000 7f1d41c06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:55.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.561+0000 7f1d41c06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:55.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.563+0000 7f1d41405640 10 monclient(hunting): handle_auth_done global_id 22768 payload 824 2024-05-04T18:18:55.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.563+0000 7f1d41405640 1 -- 172.21.15.19:0/444290993 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c140310 msgr2=0x7f1d3c0b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.563+0000 7f1d41405640 1 --2- 172.21.15.19:0/444290993 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c140310 0x7f1d3c0b76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.563+0000 7f1d41405640 1 -- 172.21.15.19:0/444290993 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c0a9fb0 msgr2=0x7f1d3c141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:55.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.563+0000 7f1d41405640 1 --2- 172.21.15.19:0/444290993 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c0a9fb0 0x7f1d3c141a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:55.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.563+0000 7f1d41405640 10 monclient: _finish_hunting 0 2024-05-04T18:18:55.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.563+0000 7f1d41405640 1 monclient: found mon.noname-b 2024-05-04T18:18:55.570 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.563+0000 7f1d41405640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:55.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.563+0000 7f1d41405640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:55.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.563+0000 7f1d41405640 1 -- 172.21.15.19:0/444290993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3c140f90 con 0x7f1d3c0b27f0 2024-05-04T18:18:55.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.565+0000 7f1d41405640 1 --2- 172.21.15.19:0/444290993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d3c0b27f0 0x7f1d3c141fb0 secure :-1 s=READY pgs=5831 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c00e850 tx=0x7f1d2c00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:55.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.565+0000 7f1d327fc640 10 client.?.objecter ms_handle_connect 0x7f1d3c0b27f0 2024-05-04T18:18:55.571 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.565+0000 7f1d327fc640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:55.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 1 -- 172.21.15.19:0/444290993 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c017070 con 0x7f1d3c0b27f0 2024-05-04T18:18:55.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:55.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:55.572 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 10 monclient: dump: 2024-05-04T18:18:55.572 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:55.572 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:55.572 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.572 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.572 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 10 monclient: _finish_auth 0 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 10 monclient: _check_auth_tickets 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 1 -- 172.21.15.19:0/444290993 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d2c00c940 con 0x7f1d3c0b27f0 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 1 -- 172.21.15.19:0/444290993 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c00cd10 con 0x7f1d3c0b27f0 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d327fc640 10 monclient: dump: 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:55.573 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d43e6a640 5 monclient: authenticate success, global_id 22768 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d43e6a640 10 monclient: _renew_subs 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d43e6a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d43e6a640 1 -- 172.21.15.19:0/444290993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d3c0b7c30 con 0x7f1d3c0b27f0 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d33fff640 4 set_mon_vals no callback set 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d43e6a640 10 client.22768.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:55.574 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d43e6a640 10 monclient: _renew_subs 2024-05-04T18:18:55.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d43e6a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:55.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d43e6a640 1 -- 172.21.15.19:0/444290993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d3c0b8130 con 0x7f1d3c0b27f0 2024-05-04T18:18:55.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.569+0000 7f1d43e6a640 1 librados: init done 2024-05-04T18:18:55.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.570+0000 7f1d43e6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:55.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.570+0000 7f1d43e6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:55.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.570+0000 7f1d43e6a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:55.575 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.570+0000 7f1d43e6a640 1 -- 172.21.15.19:0/444290993 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d3c068b10 con 0x7f1d3c0b27f0 2024-05-04T18:18:55.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.826+0000 7f1d327fc640 1 -- 172.21.15.19:0/444290993 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1d2c01a070 con 0x7f1d3c0b27f0 2024-05-04T18:18:55.827 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.826+0000 7f1d327fc640 1 --2- 172.21.15.19:0/444290993 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f1d2006d810 0x7f1d2006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:55.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.826+0000 7f1d41c06640 1 --2- 172.21.15.19:0/444290993 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f1d2006d810 0x7f1d2006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:55.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.826+0000 7f1d41c06640 10 monclient: get_auth_request con 0x7f1d2006d810 auth_method 0 2024-05-04T18:18:55.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.827+0000 7f1d327fc640 10 client.22768.objecter ms_handle_connect 0x7f1d2006d810 2024-05-04T18:18:55.828 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.827+0000 7f1d41c06640 1 --2- 172.21.15.19:0/444290993 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f1d2006d810 0x7f1d2006fcd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1d38036000 tx=0x7f1d3803d040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:55.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.831+0000 7f1d327fc640 1 -- 172.21.15.19:0/444290993 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f1d2c014de0 con 0x7f1d3c0b27f0 2024-05-04T18:18:55.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.831+0000 7f1d327fc640 10 client.22768.objecter ms_dispatch 0x7f1d3c000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:55.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.831+0000 7f1d327fc640 3 client.22768.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:55.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.831+0000 7f1d327fc640 3 client.22768.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:55.832 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.831+0000 7f1d327fc640 20 client.22768.objecter dump_active .. 0 homeless 2024-05-04T18:18:55.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.841+0000 7f1d327fc640 1 -- 172.21.15.19:0/444290993 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1d2c059890 con 0x7f1d3c0b27f0 2024-05-04T18:18:55.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.841+0000 7f1d327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:55.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:55.841+0000 7f1d327fc640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:18:56.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.036+0000 7f1d43e6a640 1 -- 172.21.15.19:0/444290993 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f1d3c0b83f0 con 0x7f1d2006d810 2024-05-04T18:18:56.037 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.036+0000 7f1d327fc640 1 -- 172.21.15.19:0/444290993 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1d3c0b83f0 con 0x7f1d2006d810 2024-05-04T18:18:56.038 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 3 to deep-scrub 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 10 librados: watch_flush enter 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 10 librados: watch_flush exit 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 20 client.22768.objecter shutdown clearing up homeless session... 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 10 client.22768.objecter successfully canceled tick 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 -- 172.21.15.19:0/444290993 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f1d2006d810 msgr2=0x7f1d2006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 --2- 172.21.15.19:0/444290993 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f1d2006d810 0x7f1d2006fcd0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1d38036000 tx=0x7f1d3803d040 comp rx=0 tx=0).stop 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 10 monclient: shutdown 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 -- 172.21.15.19:0/444290993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d3c0b27f0 msgr2=0x7f1d3c141fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 --2- 172.21.15.19:0/444290993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d3c0b27f0 0x7f1d3c141fb0 secure :-1 s=READY pgs=5831 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c00e850 tx=0x7f1d2c00ed10 comp rx=0 tx=0).stop 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 -- 172.21.15.19:0/444290993 shutdown_connections 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 --2- 172.21.15.19:0/444290993 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f1d2006d810 0x7f1d2006fcd0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 --2- 172.21.15.19:0/444290993 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f1d3c140310 0x7f1d3c0b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 --2- 172.21.15.19:0/444290993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d3c0b27f0 0x7f1d3c141fb0 unknown :-1 s=CLOSED pgs=5831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 --2- 172.21.15.19:0/444290993 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f1d3c0a9fb0 0x7f1d3c141a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 -- 172.21.15.19:0/444290993 >> 172.21.15.19:0/444290993 conn(0x7f1d3c0b4e90 msgr2=0x7f1d3c13e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:56.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 -- 172.21.15.19:0/444290993 shutdown_connections 2024-05-04T18:18:56.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 -- 172.21.15.19:0/444290993 wait complete. 2024-05-04T18:18:56.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.039+0000 7f1d43e6a640 1 librados: shutdown 2024-05-04T18:18:56.052 INFO:tasks.ceph:Scrubbing osd.4 2024-05-04T18:18:56.053 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-04T18:18:56.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.155+0000 7fc4559f7640 10 monclient: get_monmap_and_config 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.155+0000 7fc4559f7640 10 monclient: build_initial_monmap 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.155+0000 7fc4559f7640 10 monclient: monmap: 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:56.157315+0000 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:56.157315+0000 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:56.157 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:56.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 1 Processor -- start 2024-05-04T18:18:56.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 1 -- start start 2024-05-04T18:18:56.159 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 10 monclient: init 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc45006dc90 0x7fc45013fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 10 monclient(hunting): picked mon.noname-b con 0x7fc45006dc90 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fc45006e070 0x7fc450140070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 10 monclient(hunting): picked mon.noname-a con 0x7fc45006e070 addr v2:172.21.15.19:3300/0 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4501460a0 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc45006d060 con 0x7fc4501460a0 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7fc450071d90 con 0x7fc45006e070 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc450140660 con 0x7fc45006dc90 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:56.160 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.156+0000 7fc4559f7640 10 monclient(hunting): authenticate will time out at 3358.271390s 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44effd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc45006dc90 0x7fc45013fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44e7fc640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7fc45006e070 0x7fc450140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44effd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc45006dc90 0x7fc45013fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58166/0 (socket says 172.21.15.19:58166) 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44f7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fc4501460a0 0x7fc450146550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:44548/0 (socket says 172.21.15.19:44548) 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44effd640 1 -- 172.21.15.19:0/2103067998 learned_addr learned my addr 172.21.15.19:0/2103067998 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44effd640 10 monclient(hunting): get_auth_request con 0x7fc45006dc90 auth_method 0 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44effd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44e7fc640 10 monclient(hunting): get_auth_request con 0x7fc45006e070 auth_method 0 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.157+0000 7fc44e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.158+0000 7fc44effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.158+0000 7fc44effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:56.161 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.158+0000 7fc44effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:56.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.158+0000 7fc44e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:56.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.158+0000 7fc44e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:56.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.158+0000 7fc44e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:56.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.160+0000 7fc44effd640 10 monclient(hunting): handle_auth_done global_id 18815 payload 274 2024-05-04T18:18:56.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.160+0000 7fc44effd640 1 -- 172.21.15.19:0/2103067998 >> v2:172.21.15.19:3300/0 conn(0x7fc45006e070 msgr2=0x7fc450140070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:56.162 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.160+0000 7fc44effd640 1 --2- 172.21.15.19:0/2103067998 >> v2:172.21.15.19:3300/0 conn(0x7fc45006e070 0x7fc450140070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.160+0000 7fc44effd640 1 -- 172.21.15.19:0/2103067998 >> v1:172.21.15.186:6789/0 conn(0x7fc4501460a0 legacy=0x7fc450146550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.160+0000 7fc44effd640 10 monclient: _finish_hunting 0 2024-05-04T18:18:56.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.160+0000 7fc44effd640 1 monclient: found mon.noname-b 2024-05-04T18:18:56.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.160+0000 7fc44effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:56.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.160+0000 7fc44effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T18:18:56.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.160+0000 7fc44effd640 1 -- 172.21.15.19:0/2103067998 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc450073790 con 0x7fc45006dc90 2024-05-04T18:18:56.163 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.162+0000 7fc44effd640 1 --2- 172.21.15.19:0/2103067998 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc45006dc90 0x7fc45013fb30 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7fc440009870 tx=0x7fc44002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=101d99e7ce50873f server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:56.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 1 -- 172.21.15.19:0/2103067998 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc440035020 con 0x7fc45006dc90 2024-05-04T18:18:56.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:56.166 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 10 monclient: dump: 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 10 monclient: _finish_auth 0 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 10 monclient: _check_auth_tickets 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 1 -- 172.21.15.19:0/2103067998 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc440031640 con 0x7fc45006dc90 2024-05-04T18:18:56.167 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 1 -- 172.21.15.19:0/2103067998 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44002d040 con 0x7fc45006dc90 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc4559f7640 5 monclient: authenticate success, global_id 18815 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc4559f7640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc4559f7640 4 set_mon_vals no callback set 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc44dffb640 10 monclient: dump: 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:56.168 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:56.169 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:56.169 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:56.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc4559f7640 10 monclient: shutdown 2024-05-04T18:18:56.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc4559f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:56.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc4559f7640 1 -- 172.21.15.19:0/2103067998 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc45006dc90 msgr2=0x7fc45013fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.165+0000 7fc4559f7640 1 --2- 172.21.15.19:0/2103067998 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc45006dc90 0x7fc45013fb30 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7fc440009870 tx=0x7fc44002b8f0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.166+0000 7fc4559f7640 1 -- 172.21.15.19:0/2103067998 shutdown_connections 2024-05-04T18:18:56.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.166+0000 7fc4559f7640 1 --2- 172.21.15.19:0/2103067998 >> v2:172.21.15.19:3300/0 conn(0x7fc45006e070 0x7fc450140070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.166+0000 7fc4559f7640 1 --2- 172.21.15.19:0/2103067998 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc45006dc90 0x7fc45013fb30 unknown :-1 s=CLOSED pgs=3169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.172 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.166+0000 7fc4559f7640 1 -- 172.21.15.19:0/2103067998 >> 172.21.15.19:0/2103067998 conn(0x7fc4500b6980 msgr2=0x7fc4500b6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.166+0000 7fc4559f7640 1 -- 172.21.15.19:0/2103067998 shutdown_connections 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.166+0000 7fc4559f7640 1 -- 172.21.15.19:0/2103067998 wait complete. 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.166+0000 7fc4559f7640 10 monclient: build_initial_monmap 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.166+0000 7fc4559f7640 10 monclient: monmap: 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 1 librados: starting msgr at 2024-05-04T18:18:56.173 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 1 librados: starting objecter 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 1 Processor -- start 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 1 -- start start 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 1 librados: setting wanted keys 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 1 librados: calling monclient init 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 10 monclient: init 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc45006dc90 0x7fc4500aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 10 monclient(hunting): picked mon.noname-a con 0x7fc45006dc90 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc45006e070 0x7fc4501db9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 10 monclient(hunting): picked mon.noname-b con 0x7fc45006e070 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc4501460a0 0x7fc450142600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 10 monclient(hunting): picked mon.noname-c con 0x7fc4501460a0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:56.174 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc45014d5b0 con 0x7fc45006dc90 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc45014d430 con 0x7fc45006e070 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc45014d730 con 0x7fc4501460a0 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.167+0000 7fc4559f7640 10 monclient(hunting): authenticate will time out at 3358.282617s 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44f7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc4501460a0 0x7fc450142600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44e7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc45006e070 0x7fc4501db9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44effd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc45006dc90 0x7fc4500aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44f7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc4501460a0 0x7fc450142600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58174/0 (socket says 172.21.15.19:58174) 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44f7fe640 1 -- 172.21.15.19:0/1392383279 learned_addr learned my addr 172.21.15.19:0/1392383279 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44f7fe640 10 monclient(hunting): get_auth_request con 0x7fc4501460a0 auth_method 0 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:56.175 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44f7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44e7fc640 10 monclient(hunting): get_auth_request con 0x7fc45006e070 auth_method 0 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44e7fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.168+0000 7fc44e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44effd640 10 monclient(hunting): get_auth_request con 0x7fc45006dc90 auth_method 0 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44effd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44effd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:56.176 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:56.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:56.177 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.169+0000 7fc44effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:56.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.172+0000 7fc44e7fc640 10 monclient(hunting): handle_auth_done global_id 22774 payload 824 2024-05-04T18:18:56.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.172+0000 7fc44e7fc640 1 -- 172.21.15.19:0/1392383279 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc4501460a0 msgr2=0x7fc450142600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.172+0000 7fc44e7fc640 1 --2- 172.21.15.19:0/1392383279 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc4501460a0 0x7fc450142600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.172+0000 7fc44e7fc640 1 -- 172.21.15.19:0/1392383279 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc45006dc90 msgr2=0x7fc4500aef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.172+0000 7fc44e7fc640 1 --2- 172.21.15.19:0/1392383279 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc45006dc90 0x7fc4500aef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.172+0000 7fc44e7fc640 10 monclient: _finish_hunting 0 2024-05-04T18:18:56.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.172+0000 7fc44e7fc640 1 monclient: found mon.noname-b 2024-05-04T18:18:56.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.172+0000 7fc44e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:56.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.172+0000 7fc44e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:56.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.172+0000 7fc44e7fc640 1 -- 172.21.15.19:0/1392383279 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4501db4c0 con 0x7fc45006e070 2024-05-04T18:18:56.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.174+0000 7fc44e7fc640 1 --2- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc45006e070 0x7fc4501db9e0 secure :-1 s=READY pgs=5833 cs=0 l=1 rev1=1 crypto rx=0x7fc44400e820 tx=0x7fc44400ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:56.178 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.174+0000 7fc437fff640 10 client.?.objecter ms_handle_connect 0x7fc45006e070 2024-05-04T18:18:56.179 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.174+0000 7fc437fff640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:56.181 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 1 -- 172.21.15.19:0/1392383279 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc444017070 con 0x7fc45006e070 2024-05-04T18:18:56.181 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:56.181 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:56.181 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 10 monclient: dump: 2024-05-04T18:18:56.181 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:56.181 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:56.181 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.181 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.181 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 10 monclient: _finish_auth 0 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 10 monclient: _check_auth_tickets 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 1 -- 172.21.15.19:0/1392383279 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc44400c940 con 0x7fc45006e070 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 1 -- 172.21.15.19:0/1392383279 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44400cd10 con 0x7fc45006e070 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc4559f7640 5 monclient: authenticate success, global_id 22774 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc4559f7640 10 monclient: _renew_subs 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc4559f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:56.182 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc4559f7640 1 -- 172.21.15.19:0/1392383279 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc450141b20 con 0x7fc45006e070 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc437fff640 10 monclient: dump: 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc44d7fa640 4 set_mon_vals no callback set 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc4559f7640 10 client.22774.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:56.183 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc4559f7640 10 monclient: _renew_subs 2024-05-04T18:18:56.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc4559f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:56.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc4559f7640 1 -- 172.21.15.19:0/1392383279 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4501dc100 con 0x7fc45006e070 2024-05-04T18:18:56.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.179+0000 7fc4559f7640 1 librados: init done 2024-05-04T18:18:56.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.180+0000 7fc4559f7640 10 client.22774.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:56.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.180+0000 7fc4559f7640 15 client.22774.objecter _session_command_op_assign -1 1 2024-05-04T18:18:56.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.180+0000 7fc4559f7640 20 client.22774.objecter _get_session osd=-1 returning homeless 2024-05-04T18:18:56.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.180+0000 7fc4559f7640 10 client.22774.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:56.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.180+0000 7fc4559f7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:56.184 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.180+0000 7fc4559f7640 1 -- 172.21.15.19:0/1392383279 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc450068b10 con 0x7fc45006e070 2024-05-04T18:18:56.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.199+0000 7fc437fff640 1 -- 172.21.15.19:0/1392383279 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc44401a070 con 0x7fc45006e070 2024-05-04T18:18:56.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.199+0000 7fc437fff640 1 --2- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc43c06d840 0x7fc43c06fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:56.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.200+0000 7fc44effd640 1 --2- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc43c06d840 0x7fc43c06fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:56.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.200+0000 7fc44effd640 10 monclient: get_auth_request con 0x7fc43c06d840 auth_method 0 2024-05-04T18:18:56.201 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.200+0000 7fc44effd640 1 --2- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc43c06d840 0x7fc43c06fd00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc440036000 tx=0x7fc44003c040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:56.202 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.200+0000 7fc437fff640 10 client.22774.objecter ms_handle_connect 0x7fc43c06d840 2024-05-04T18:18:56.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.202+0000 7fc437fff640 1 -- 172.21.15.19:0/1392383279 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7fc444014de0 con 0x7fc45006e070 2024-05-04T18:18:56.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.202+0000 7fc437fff640 10 client.22774.objecter ms_dispatch 0x7fc450000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:56.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.202+0000 7fc437fff640 3 client.22774.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:56.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.202+0000 7fc437fff640 3 client.22774.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:56.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc437fff640 10 client.22774.objecter checking command 1 2024-05-04T18:18:56.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc437fff640 1 --2- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] conn(0x7fc43c0756a0 0x7fc43c077b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:56.204 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc437fff640 20 client.22774.objecter _get_session s=0x7fc43c074fd0 osd=4 3 2024-05-04T18:18:56.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc437fff640 20 client.22774.objecter put_session s=0x7fc43c074fd0 osd=4 3 2024-05-04T18:18:56.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc437fff640 15 client.22774.objecter _session_command_op_remove -1 1 2024-05-04T18:18:56.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc437fff640 20 client.22774.objecter _get_session s=0x7fc43c074fd0 osd=4 3 2024-05-04T18:18:56.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc437fff640 20 client.22774.objecter get_session s=0x7fc43c074fd0 osd=4 3 2024-05-04T18:18:56.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc437fff640 15 client.22774.objecter _session_command_op_assign 4 1 2024-05-04T18:18:56.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc437fff640 20 client.22774.objecter put_session s=0x7fc43c074fd0 osd=4 4 2024-05-04T18:18:56.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc437fff640 10 client.22774.objecter _send_command 1 2024-05-04T18:18:56.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc437fff640 1 -- 172.21.15.19:0/1392383279 --> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc43c078210 con 0x7fc43c0756a0 2024-05-04T18:18:56.205 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc437fff640 20 client.22774.objecter dump_active .. 0 homeless 2024-05-04T18:18:56.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.203+0000 7fc44f7fe640 1 --2- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] conn(0x7fc43c0756a0 0x7fc43c077b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:56.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.204+0000 7fc44f7fe640 10 monclient: get_auth_request con 0x7fc43c0756a0 auth_method 0 2024-05-04T18:18:56.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.205+0000 7fc437fff640 1 -- 172.21.15.19:0/1392383279 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=1186) ==== 24+0+0 (secure 0 0 0) 0x7fc444055630 con 0x7fc45006e070 2024-05-04T18:18:56.206 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.205+0000 7fc437fff640 10 monclient: handle_get_version_reply finishing 1 version 1186 2024-05-04T18:18:56.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.207+0000 7fc437fff640 10 client.22774.objecter ms_handle_connect 0x7fc43c0756a0 2024-05-04T18:18:56.208 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.207+0000 7fc44f7fe640 1 --2- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] conn(0x7fc43c0756a0 0x7fc43c077b40 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:56.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.232+0000 7fc437fff640 1 -- 172.21.15.19:0/1392383279 <== osd.4 v2:172.21.15.186:6818/356995042 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc43c078210 con 0x7fc43c0756a0 2024-05-04T18:18:56.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.232+0000 7fc437fff640 10 client.22774.objecter ms_dispatch 0x7fc450000b90 command_reply(tid 1: 0 ) 2024-05-04T18:18:56.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.232+0000 7fc437fff640 10 client.22774.objecter _finish_command 1 = system:0 2024-05-04T18:18:56.233 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.232+0000 7fc437fff640 20 client.22774.objecter put_session s=0x7fc43c074fd0 osd=4 4 2024-05-04T18:18:56.234 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.232+0000 7fc437fff640 15 client.22774.objecter _session_command_op_remove 4 1 2024-05-04T18:18:56.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.250+0000 7fc4559f7640 10 client.22774.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-04T18:18:56.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.250+0000 7fc4559f7640 15 client.22774.objecter _session_command_op_assign -1 2 2024-05-04T18:18:56.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.250+0000 7fc4559f7640 20 client.22774.objecter _get_session s=0x7fc43c074fd0 osd=4 3 2024-05-04T18:18:56.253 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.250+0000 7fc4559f7640 20 client.22774.objecter put_session s=0x7fc43c074fd0 osd=4 3 2024-05-04T18:18:56.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.250+0000 7fc4559f7640 20 client.22774.objecter _get_session s=0x7fc43c074fd0 osd=4 3 2024-05-04T18:18:56.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.250+0000 7fc4559f7640 15 client.22774.objecter _session_command_op_remove -1 2 2024-05-04T18:18:56.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.250+0000 7fc4559f7640 20 client.22774.objecter get_session s=0x7fc43c074fd0 osd=4 3 2024-05-04T18:18:56.254 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.250+0000 7fc4559f7640 15 client.22774.objecter _session_command_op_assign 4 2 2024-05-04T18:18:56.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.250+0000 7fc4559f7640 20 client.22774.objecter put_session s=0x7fc43c074fd0 osd=4 4 2024-05-04T18:18:56.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.250+0000 7fc4559f7640 10 client.22774.objecter _send_command 2 2024-05-04T18:18:56.255 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.250+0000 7fc4559f7640 1 -- 172.21.15.19:0/1392383279 --> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc450142d90 con 0x7fc43c0756a0 2024-05-04T18:18:56.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.263+0000 7fc437fff640 1 -- 172.21.15.19:0/1392383279 <== osd.4 v2:172.21.15.186:6818/356995042 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc450142d90 con 0x7fc43c0756a0 2024-05-04T18:18:56.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.263+0000 7fc437fff640 10 client.22774.objecter ms_dispatch 0x7fc450000b90 command_reply(tid 2: 0 ) 2024-05-04T18:18:56.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.263+0000 7fc437fff640 10 client.22774.objecter _finish_command 2 = system:0 2024-05-04T18:18:56.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.263+0000 7fc437fff640 20 client.22774.objecter put_session s=0x7fc43c074fd0 osd=4 4 2024-05-04T18:18:56.265 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.263+0000 7fc437fff640 15 client.22774.objecter _session_command_op_remove 4 2 2024-05-04T18:18:56.265 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-05-04T18:18:56.265 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T18:18:56.265 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.263+0000 7fc4559f7640 10 librados: watch_flush enter 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 10 librados: watch_flush exit 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 10 client.22774.objecter close_session for osd.4 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 1 -- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] conn(0x7fc43c0756a0 msgr2=0x7fc43c077b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 1 --2- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:6818/356995042,v1:172.21.15.186:6819/356995042] conn(0x7fc43c0756a0 0x7fc43c077b40 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 20 client.22774.objecter put_session s=0x7fc43c074fd0 osd=4 1 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 20 client.22774.objecter shutdown clearing up homeless session... 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 10 client.22774.objecter successfully canceled tick 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 1 -- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc43c06d840 msgr2=0x7fc43c06fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 1 --2- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc43c06d840 0x7fc43c06fd00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc440036000 tx=0x7fc44003c040 comp rx=0 tx=0).stop 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 10 monclient: shutdown 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:56.267 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 1 -- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc45006e070 msgr2=0x7fc4501db9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 1 --2- 172.21.15.19:0/1392383279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc45006e070 0x7fc4501db9e0 secure :-1 s=READY pgs=5833 cs=0 l=1 rev1=1 crypto rx=0x7fc44400e820 tx=0x7fc44400ece0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc44f7fe640 1 -- 172.21.15.19:0/1392383279 reap_dead start 2024-05-04T18:18:56.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 1 -- 172.21.15.19:0/1392383279 shutdown_connections 2024-05-04T18:18:56.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 1 -- 172.21.15.19:0/1392383279 >> 172.21.15.19:0/1392383279 conn(0x7fc4500b6980 msgr2=0x7fc4500b5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:56.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 1 -- 172.21.15.19:0/1392383279 shutdown_connections 2024-05-04T18:18:56.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 1 -- 172.21.15.19:0/1392383279 wait complete. 2024-05-04T18:18:56.268 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.264+0000 7fc4559f7640 1 librados: shutdown 2024-05-04T18:18:56.276 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-04T18:18:56.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.379+0000 7f96b1faf640 10 monclient: get_monmap_and_config 2024-05-04T18:18:56.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.379+0000 7f96b1faf640 10 monclient: build_initial_monmap 2024-05-04T18:18:56.381 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.379+0000 7f96b1faf640 10 monclient: monmap: 2024-05-04T18:18:56.381 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:56.381 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:56.381 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:56.381516+0000 2024-05-04T18:18:56.381 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:56.381516+0000 2024-05-04T18:18:56.381 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:56.381 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:56.381 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:56.381 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:56.381 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:56.382 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:56.382 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:56.382 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:56.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.380+0000 7f96b1faf640 1 Processor -- start 2024-05-04T18:18:56.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.380+0000 7f96b1faf640 1 -- start start 2024-05-04T18:18:56.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.380+0000 7f96b1faf640 10 monclient: init 2024-05-04T18:18:56.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.381+0000 7f96b1faf640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:56.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.381+0000 7f96b1faf640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-05-04T18:18:56.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.381+0000 7f96b1faf640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f96ac06dc90 0x7f96ac13fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:56.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.381+0000 7f96b1faf640 10 monclient(hunting): picked mon.noname-b con 0x7f96ac06dc90 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:56.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.381+0000 7f96b1faf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f96ac06e070 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:56.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.381+0000 7f96b1faf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96ac1467b0 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:56.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.381+0000 7f96b1faf640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96ac06d060 con 0x7f96ac1467b0 2024-05-04T18:18:56.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.381+0000 7f96b1faf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96ac071d90 con 0x7f96ac06e070 2024-05-04T18:18:56.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.381+0000 7f96b1faf640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:56.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.381+0000 7f96b1faf640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f96ac140660 con 0x7f96ac06dc90 2024-05-04T18:18:56.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.381+0000 7f96b1faf640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:56.384 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.381+0000 7f96b1faf640 10 monclient(hunting): authenticate will time out at 3358.495934s 2024-05-04T18:18:56.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.432+0000 7f96abfff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f96ac1467b0 0x7f96ac14d430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:53114/0 (socket says 172.21.15.19:53114) 2024-05-04T18:18:56.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.432+0000 7f96abfff640 1 -- 172.21.15.19:0/1888807535 learned_addr learned my addr 172.21.15.19:0/1888807535 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T18:18:56.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.432+0000 7f96ab7fe640 1 --2- 172.21.15.19:0/1888807535 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f96ac06dc90 0x7f96ac13fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:56.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.432+0000 7f96ab7fe640 10 monclient(hunting): get_auth_request con 0x7f96ac06dc90 auth_method 0 2024-05-04T18:18:56.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.432+0000 7f96ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:56.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.432+0000 7f96ab7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:56.434 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.432+0000 7f96ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:56.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.433+0000 7f96ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:56.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.433+0000 7f96ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:56.435 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.433+0000 7f96ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:56.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.435+0000 7f96ab7fe640 10 monclient(hunting): handle_auth_done global_id 18821 payload 274 2024-05-04T18:18:56.436 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.435+0000 7f96ab7fe640 1 -- 172.21.15.19:0/1888807535 >> v1:172.21.15.186:6789/0 conn(0x7f96ac06e070 legacy=0x7f96ac1460a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:56.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.435+0000 7f96ab7fe640 1 -- 172.21.15.19:0/1888807535 >> v1:172.21.15.19:6789/0 conn(0x7f96ac1467b0 legacy=0x7f96ac14d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.435+0000 7f96ab7fe640 10 monclient: _finish_hunting 0 2024-05-04T18:18:56.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.435+0000 7f96ab7fe640 1 monclient: found mon.noname-b 2024-05-04T18:18:56.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.435+0000 7f96ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:56.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.435+0000 7f96ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T18:18:56.437 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.435+0000 7f96ab7fe640 1 -- 172.21.15.19:0/1888807535 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96ac073790 con 0x7f96ac06dc90 2024-05-04T18:18:56.443 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.441+0000 7f96ab7fe640 1 --2- 172.21.15.19:0/1888807535 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f96ac06dc90 0x7f96ac13fb30 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7f969c0097d0 tx=0x7f969c02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2711fac0fe83ea2a server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:56.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 1 -- 172.21.15.19:0/1888807535 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c031770 con 0x7f96ac06dc90 2024-05-04T18:18:56.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:56.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:56.446 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 10 monclient: dump: 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 10 monclient: _finish_auth 0 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 10 monclient: _check_auth_tickets 2024-05-04T18:18:56.447 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 1 -- 172.21.15.19:0/1888807535 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f969c037070 con 0x7f96ac06dc90 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 1 -- 172.21.15.19:0/1888807535 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969c031b30 con 0x7f96ac06dc90 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96aa7fc640 10 monclient: dump: 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:56.448 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:56.449 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:56.449 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:56.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96b1faf640 5 monclient: authenticate success, global_id 18821 2024-05-04T18:18:56.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96b1faf640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:56.449 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96b1faf640 4 set_mon_vals no callback set 2024-05-04T18:18:56.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96b1faf640 10 monclient: shutdown 2024-05-04T18:18:56.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96b1faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:56.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96b1faf640 1 -- 172.21.15.19:0/1888807535 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f96ac06dc90 msgr2=0x7f96ac13fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.445+0000 7f96b1faf640 1 --2- 172.21.15.19:0/1888807535 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f96ac06dc90 0x7f96ac13fb30 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7f969c0097d0 tx=0x7f969c02b810 comp rx=0 tx=0).stop 2024-05-04T18:18:56.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.446+0000 7f96b1faf640 1 -- 172.21.15.19:0/1888807535 shutdown_connections 2024-05-04T18:18:56.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.446+0000 7f96b1faf640 1 --2- 172.21.15.19:0/1888807535 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f96ac06dc90 0x7f96ac13fb30 unknown :-1 s=CLOSED pgs=3170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.446+0000 7f96b1faf640 1 -- 172.21.15.19:0/1888807535 >> 172.21.15.19:0/1888807535 conn(0x7f96ac0b6980 msgr2=0x7f96ac0b6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:56.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.446+0000 7f96b1faf640 1 -- 172.21.15.19:0/1888807535 shutdown_connections 2024-05-04T18:18:56.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.446+0000 7f96b1faf640 1 -- 172.21.15.19:0/1888807535 wait complete. 2024-05-04T18:18:56.452 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.447+0000 7f96b1faf640 10 monclient: build_initial_monmap 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.447+0000 7f96b1faf640 10 monclient: monmap: 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.447+0000 7f96b1faf640 1 librados: starting msgr at 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.447+0000 7f96b1faf640 1 librados: starting objecter 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.447+0000 7f96b1faf640 1 Processor -- start 2024-05-04T18:18:56.453 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.447+0000 7f96b1faf640 1 -- start start 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.447+0000 7f96b1faf640 1 librados: setting wanted keys 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.447+0000 7f96b1faf640 1 librados: calling monclient init 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.447+0000 7f96b1faf640 10 monclient: init 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96ac06dc90 0x7f96ac0af270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 10 monclient(hunting): picked mon.noname-a con 0x7f96ac06dc90 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f96ac06e070 0x7f96ac141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 10 monclient(hunting): picked mon.noname-c con 0x7f96ac06e070 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96ac1467b0 0x7f96ac141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 10 monclient(hunting): picked mon.noname-b con 0x7f96ac1467b0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f96ac1401f0 con 0x7f96ac06dc90 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:56.454 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f96ac140070 con 0x7f96ac1467b0 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f96ac140370 con 0x7f96ac06e070 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96b1faf640 10 monclient(hunting): authenticate will time out at 3358.563033s 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96abfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96ac1467b0 0x7f96ac141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.448+0000 7f96aaffd640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f96ac06e070 0x7f96ac141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96abfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96ac1467b0 0x7f96ac141660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:45416/0 (socket says 172.21.15.19:45416) 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96abfff640 1 -- 172.21.15.19:0/579938461 learned_addr learned my addr 172.21.15.19:0/579938461 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96abfff640 10 monclient(hunting): get_auth_request con 0x7f96ac1467b0 auth_method 0 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96abfff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96abfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96aaffd640 10 monclient(hunting): get_auth_request con 0x7f96ac06e070 auth_method 0 2024-05-04T18:18:56.455 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:56.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96aaffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:56.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:56.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:56.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:56.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.449+0000 7f96abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:56.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.450+0000 7f96aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:56.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.450+0000 7f96aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:56.456 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.450+0000 7f96aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:56.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.452+0000 7f96abfff640 10 monclient(hunting): handle_auth_done global_id 22777 payload 824 2024-05-04T18:18:56.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.452+0000 7f96abfff640 1 -- 172.21.15.19:0/579938461 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f96ac06e070 msgr2=0x7f96ac141120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.452+0000 7f96abfff640 1 --2- 172.21.15.19:0/579938461 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f96ac06e070 0x7f96ac141120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.452+0000 7f96abfff640 1 -- 172.21.15.19:0/579938461 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96ac06dc90 msgr2=0x7f96ac0af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.452+0000 7f96abfff640 1 --2- 172.21.15.19:0/579938461 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96ac06dc90 0x7f96ac0af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.452+0000 7f96abfff640 10 monclient: _finish_hunting 0 2024-05-04T18:18:56.457 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.452+0000 7f96abfff640 1 monclient: found mon.noname-b 2024-05-04T18:18:56.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.452+0000 7f96abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:56.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.452+0000 7f96abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:56.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.452+0000 7f96abfff640 1 -- 172.21.15.19:0/579938461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96ac0b0300 con 0x7f96ac1467b0 2024-05-04T18:18:56.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.455+0000 7f96abfff640 1 --2- 172.21.15.19:0/579938461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96ac1467b0 0x7f96ac141660 secure :-1 s=READY pgs=5834 cs=0 l=1 rev1=1 crypto rx=0x7f96a400c9f0 tx=0x7f96a400ceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:56.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.455+0000 7f9693fff640 10 client.?.objecter ms_handle_connect 0x7f96ac1467b0 2024-05-04T18:18:56.458 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.455+0000 7f9693fff640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 1 -- 172.21.15.19:0/579938461 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a4017070 con 0x7f96ac1467b0 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 10 monclient: dump: 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:56.461 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 10 monclient: _finish_auth 0 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 10 monclient: _check_auth_tickets 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 1 -- 172.21.15.19:0/579938461 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a400ab70 con 0x7f96ac1467b0 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 1 -- 172.21.15.19:0/579938461 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a4012400 con 0x7f96ac1467b0 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f9693fff640 10 monclient: dump: 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:56.462 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f96b1faf640 5 monclient: authenticate success, global_id 22777 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f96b1faf640 10 monclient: _renew_subs 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f96a9ffb640 4 set_mon_vals no callback set 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f96b1faf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.458+0000 7f96b1faf640 1 -- 172.21.15.19:0/579938461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96ac0b05e0 con 0x7f96ac1467b0 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.459+0000 7f96b1faf640 10 client.22777.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.459+0000 7f96b1faf640 10 monclient: _renew_subs 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.459+0000 7f96b1faf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.459+0000 7f96b1faf640 1 -- 172.21.15.19:0/579938461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96ac140920 con 0x7f96ac1467b0 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.459+0000 7f96b1faf640 1 librados: init done 2024-05-04T18:18:56.463 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.459+0000 7f96b1faf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:56.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.459+0000 7f96b1faf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:56.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.459+0000 7f96b1faf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:56.464 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.459+0000 7f96b1faf640 1 -- 172.21.15.19:0/579938461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96ac068b10 con 0x7f96ac1467b0 2024-05-04T18:18:56.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.479+0000 7f9693fff640 1 -- 172.21.15.19:0/579938461 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f96a401a070 con 0x7f96ac1467b0 2024-05-04T18:18:56.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.479+0000 7f9693fff640 1 --2- 172.21.15.19:0/579938461 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f969806d810 0x7f969806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:56.480 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.479+0000 7f96ab7fe640 1 --2- 172.21.15.19:0/579938461 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f969806d810 0x7f969806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:56.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.479+0000 7f96ab7fe640 10 monclient: get_auth_request con 0x7f969806d810 auth_method 0 2024-05-04T18:18:56.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.480+0000 7f9693fff640 10 client.22777.objecter ms_handle_connect 0x7f969806d810 2024-05-04T18:18:56.481 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.480+0000 7f96ab7fe640 1 --2- 172.21.15.19:0/579938461 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f969806d810 0x7f969806fcd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f969c036000 tx=0x7f969c030000 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:56.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.482+0000 7f9693fff640 1 -- 172.21.15.19:0/579938461 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f96a4057360 con 0x7f96ac1467b0 2024-05-04T18:18:56.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.482+0000 7f9693fff640 10 client.22777.objecter ms_dispatch 0x7f96ac000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:56.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.482+0000 7f9693fff640 3 client.22777.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:56.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.482+0000 7f9693fff640 3 client.22777.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:56.483 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.482+0000 7f9693fff640 20 client.22777.objecter dump_active .. 0 homeless 2024-05-04T18:18:56.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.733+0000 7f9693fff640 1 -- 172.21.15.19:0/579938461 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f96a40592c0 con 0x7f96ac1467b0 2024-05-04T18:18:56.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.733+0000 7f9693fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:56.734 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.733+0000 7f9693fff640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:18:56.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.927+0000 7f96b1faf640 1 -- 172.21.15.19:0/579938461 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f96ac0b5240 con 0x7f969806d810 2024-05-04T18:18:56.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.928+0000 7f9693fff640 1 -- 172.21.15.19:0/579938461 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f96ac0b5240 con 0x7f969806d810 2024-05-04T18:18:56.929 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 4 to deep-scrub 2024-05-04T18:18:56.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 10 librados: watch_flush enter 2024-05-04T18:18:56.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 10 librados: watch_flush exit 2024-05-04T18:18:56.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 20 client.22777.objecter shutdown clearing up homeless session... 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 10 client.22777.objecter successfully canceled tick 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 -- 172.21.15.19:0/579938461 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f969806d810 msgr2=0x7f969806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 --2- 172.21.15.19:0/579938461 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f969806d810 0x7f969806fcd0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f969c036000 tx=0x7f969c030000 comp rx=0 tx=0).stop 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 10 monclient: shutdown 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 -- 172.21.15.19:0/579938461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96ac1467b0 msgr2=0x7f96ac141660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 --2- 172.21.15.19:0/579938461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96ac1467b0 0x7f96ac141660 secure :-1 s=READY pgs=5834 cs=0 l=1 rev1=1 crypto rx=0x7f96a400c9f0 tx=0x7f96a400ceb0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 -- 172.21.15.19:0/579938461 shutdown_connections 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 --2- 172.21.15.19:0/579938461 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f969806d810 0x7f969806fcd0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 --2- 172.21.15.19:0/579938461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96ac1467b0 0x7f96ac141660 unknown :-1 s=CLOSED pgs=5834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 --2- 172.21.15.19:0/579938461 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f96ac06e070 0x7f96ac141120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 --2- 172.21.15.19:0/579938461 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f96ac06dc90 0x7f96ac0af270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 -- 172.21.15.19:0/579938461 >> 172.21.15.19:0/579938461 conn(0x7f96ac0b6980 msgr2=0x7f96ac0b6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:56.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 -- 172.21.15.19:0/579938461 shutdown_connections 2024-05-04T18:18:56.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 -- 172.21.15.19:0/579938461 wait complete. 2024-05-04T18:18:56.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:56.930+0000 7f96b1faf640 1 librados: shutdown 2024-05-04T18:18:56.943 INFO:tasks.ceph:Scrubbing osd.5 2024-05-04T18:18:56.943 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-04T18:18:57.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.048+0000 7fc620e25640 10 monclient: get_monmap_and_config 2024-05-04T18:18:57.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.048+0000 7fc620e25640 10 monclient: build_initial_monmap 2024-05-04T18:18:57.050 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.048+0000 7fc620e25640 10 monclient: monmap: 2024-05-04T18:18:57.050 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:57.050 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.050 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:57.050752+0000 2024-05-04T18:18:57.050 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:57.050752+0000 2024-05-04T18:18:57.050 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:57.051 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.051 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:57.051 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:57.051 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:57.051 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:57.051 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:57.051 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.049+0000 7fc620e25640 1 Processor -- start 2024-05-04T18:18:57.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.049+0000 7fc620e25640 1 -- start start 2024-05-04T18:18:57.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.049+0000 7fc620e25640 10 monclient: init 2024-05-04T18:18:57.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc620e25640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:57.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc620e25640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-05-04T18:18:57.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc620e25640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc61c071b20 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:57.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc620e25640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc61c13fcc0 0x7fc61c140120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc620e25640 10 monclient(hunting): picked mon.noname-b con 0x7fc61c13fcc0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:57.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc620e25640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc61c1430d0 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:57.053 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc620e25640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc61c071120 con 0x7fc61c1430d0 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc620e25640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc61c071390 con 0x7fc61c071b20 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc620e25640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc620e25640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc61c072610 con 0x7fc61c13fcc0 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc620e25640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc620e25640 10 monclient(hunting): authenticate will time out at 3359.165018s 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.050+0000 7fc619d9b640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc61c13fcc0 0x7fc61c140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.051+0000 7fc61ad9d640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fc61c1430d0 0x7fc61c143540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:53130/0 (socket says 172.21.15.19:53130) 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.051+0000 7fc61ad9d640 1 -- 172.21.15.19:0/3782961890 learned_addr learned my addr 172.21.15.19:0/3782961890 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.051+0000 7fc619d9b640 10 monclient(hunting): get_auth_request con 0x7fc61c13fcc0 auth_method 0 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.051+0000 7fc619d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.051+0000 7fc619d9b640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.051+0000 7fc619d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.052+0000 7fc619d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.054 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.052+0000 7fc619d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.055 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.052+0000 7fc619d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc61959a640 1 -- 172.21.15.19:0/3782961890 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 693455589 0 0) 0x7fc61c071390 con 0x7fc61c071b20 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc61959a640 10 monclient(hunting): _negotiate 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc61959a640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc61959a640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc61959a640 10 monclient(hunting): my global_id is 22780 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc61959a640 1 -- 172.21.15.19:0/3782961890 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc608003520 con 0x7fc61c071b20 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc61959a640 1 -- 172.21.15.19:0/3782961890 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1625957625 0 0) 0x7fc61c071120 con 0x7fc61c1430d0 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc61959a640 10 monclient(hunting): _negotiate 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc61959a640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc61959a640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc61959a640 10 monclient(hunting): my global_id is 15816 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc61959a640 1 -- 172.21.15.19:0/3782961890 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc61c071390 con 0x7fc61c1430d0 2024-05-04T18:18:57.056 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc619d9b640 10 monclient(hunting): handle_auth_done global_id 18827 payload 274 2024-05-04T18:18:57.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc619d9b640 1 -- 172.21.15.19:0/3782961890 >> v1:172.21.15.186:6789/0 conn(0x7fc61c071b20 legacy=0x7fc61c071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc619d9b640 1 -- 172.21.15.19:0/3782961890 >> v1:172.21.15.19:6789/0 conn(0x7fc61c1430d0 legacy=0x7fc61c143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc619d9b640 10 monclient: _finish_hunting 0 2024-05-04T18:18:57.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc619d9b640 1 monclient: found mon.noname-b 2024-05-04T18:18:57.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc619d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:57.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc619d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T18:18:57.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.053+0000 7fc619d9b640 1 -- 172.21.15.19:0/3782961890 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c06c4c0 con 0x7fc61c13fcc0 2024-05-04T18:18:57.057 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.056+0000 7fc619d9b640 1 --2- 172.21.15.19:0/3782961890 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc61c13fcc0 0x7fc61c140120 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7fc60c0097d0 tx=0x7fc60c02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f1e48a5ddf3a5d server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:57.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 1 -- 172.21.15.19:0/3782961890 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c034070 con 0x7fc61c13fcc0 2024-05-04T18:18:57.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:57.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:57.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 10 monclient: dump: 2024-05-04T18:18:57.061 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:57.061 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.061 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.061 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.061 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:57.061 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 10 monclient: _finish_auth 0 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 10 monclient: _check_auth_tickets 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 1 -- 172.21.15.19:0/3782961890 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc60c030880 con 0x7fc61c13fcc0 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 1 -- 172.21.15.19:0/3782961890 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c030c50 con 0x7fc61c13fcc0 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc61959a640 10 monclient: dump: 2024-05-04T18:18:57.062 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:57.063 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.063 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.063 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.063 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:57.063 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.063 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:57.063 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:57.063 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:57.063 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc620e25640 5 monclient: authenticate success, global_id 18827 2024-05-04T18:18:57.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc620e25640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:57.063 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.059+0000 7fc620e25640 4 set_mon_vals no callback set 2024-05-04T18:18:57.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.060+0000 7fc620e25640 10 monclient: shutdown 2024-05-04T18:18:57.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.060+0000 7fc620e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:57.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.060+0000 7fc620e25640 1 -- 172.21.15.19:0/3782961890 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc61c13fcc0 msgr2=0x7fc61c140120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.060+0000 7fc620e25640 1 --2- 172.21.15.19:0/3782961890 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc61c13fcc0 0x7fc61c140120 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7fc60c0097d0 tx=0x7fc60c02b9c0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.066 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.060+0000 7fc620e25640 1 -- 172.21.15.19:0/3782961890 shutdown_connections 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.060+0000 7fc620e25640 1 --2- 172.21.15.19:0/3782961890 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc61c13fcc0 0x7fc61c140120 unknown :-1 s=CLOSED pgs=3171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.060+0000 7fc620e25640 1 -- 172.21.15.19:0/3782961890 >> 172.21.15.19:0/3782961890 conn(0x7fc61c0679a0 msgr2=0x7fc61c0b6530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.060+0000 7fc620e25640 1 -- 172.21.15.19:0/3782961890 shutdown_connections 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.060+0000 7fc620e25640 1 -- 172.21.15.19:0/3782961890 wait complete. 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.061+0000 7fc620e25640 10 monclient: build_initial_monmap 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.061+0000 7fc620e25640 10 monclient: monmap: 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:57.067 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.061+0000 7fc620e25640 1 librados: starting msgr at 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.061+0000 7fc620e25640 1 librados: starting objecter 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 1 Processor -- start 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 1 -- start start 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 1 librados: setting wanted keys 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 1 librados: calling monclient init 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 10 monclient: init 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc61c071b20 0x7fc61c1d84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 10 monclient(hunting): picked mon.noname-b con 0x7fc61c071b20 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc61c13fcc0 0x7fc61c0aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 10 monclient(hunting): picked mon.noname-a con 0x7fc61c13fcc0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc61c1430d0 0x7fc61c0af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.068 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 10 monclient(hunting): picked mon.noname-c con 0x7fc61c1430d0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c14e800 con 0x7fc61c13fcc0 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c14e680 con 0x7fc61c071b20 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc61c14e980 con 0x7fc61c1430d0 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.062+0000 7fc620e25640 10 monclient(hunting): authenticate will time out at 3359.177349s 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61ad9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc61c1430d0 0x7fc61c0af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc619d9b640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc61c13fcc0 0x7fc61c0aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc61c071b20 0x7fc61c1d84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61ad9d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc61c1430d0 0x7fc61c0af0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58222/0 (socket says 172.21.15.19:58222) 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61ad9d640 1 -- 172.21.15.19:0/978704061 learned_addr learned my addr 172.21.15.19:0/978704061 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61ad9d640 10 monclient(hunting): get_auth_request con 0x7fc61c1430d0 auth_method 0 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.069 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61ad9d640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc619d9b640 10 monclient(hunting): get_auth_request con 0x7fc61c13fcc0 auth_method 0 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc619d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc619d9b640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc619d9b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61a59c640 10 monclient(hunting): get_auth_request con 0x7fc61c071b20 auth_method 0 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61a59c640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61a59c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.063+0000 7fc61ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.064+0000 7fc61ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.064+0000 7fc61a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.070 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.064+0000 7fc61a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.064+0000 7fc61a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.064+0000 7fc619d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.064+0000 7fc619d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.071 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.064+0000 7fc619d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.066+0000 7fc61a59c640 10 monclient(hunting): handle_auth_done global_id 22783 payload 824 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.066+0000 7fc61a59c640 1 -- 172.21.15.19:0/978704061 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc61c1430d0 msgr2=0x7fc61c0af0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.066+0000 7fc61a59c640 1 --2- 172.21.15.19:0/978704061 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc61c1430d0 0x7fc61c0af0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.066+0000 7fc61a59c640 1 -- 172.21.15.19:0/978704061 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc61c13fcc0 msgr2=0x7fc61c0aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.066+0000 7fc61a59c640 1 --2- 172.21.15.19:0/978704061 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc61c13fcc0 0x7fc61c0aeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.066+0000 7fc61a59c640 10 monclient: _finish_hunting 0 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.066+0000 7fc61a59c640 1 monclient: found mon.noname-b 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.066+0000 7fc61a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.066+0000 7fc61a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.066+0000 7fc61a59c640 1 -- 172.21.15.19:0/978704061 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c0adf80 con 0x7fc61c071b20 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.069+0000 7fc61a59c640 1 --2- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc61c071b20 0x7fc61c1d84c0 secure :-1 s=READY pgs=5836 cs=0 l=1 rev1=1 crypto rx=0x7fc60400c3c0 tx=0x7fc60400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.069+0000 7fc612ffd640 10 client.?.objecter ms_handle_connect 0x7fc61c071b20 2024-05-04T18:18:57.072 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.069+0000 7fc612ffd640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:57.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 1 -- 172.21.15.19:0/978704061 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604017070 con 0x7fc61c071b20 2024-05-04T18:18:57.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:57.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:57.074 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 10 monclient: dump: 2024-05-04T18:18:57.074 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:57.074 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.074 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 10 monclient: _finish_auth 0 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 10 monclient: _check_auth_tickets 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 1 -- 172.21.15.19:0/978704061 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6040052d0 con 0x7fc61c071b20 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 1 -- 172.21.15.19:0/978704061 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604003810 con 0x7fc61c071b20 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:57.075 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc612ffd640 10 monclient: dump: 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc620e25640 5 monclient: authenticate success, global_id 22783 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc620e25640 10 monclient: _renew_subs 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc620e25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.076 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc620e25640 1 -- 172.21.15.19:0/978704061 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c13d750 con 0x7fc61c071b20 2024-05-04T18:18:57.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc618d99640 4 set_mon_vals no callback set 2024-05-04T18:18:57.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc620e25640 10 client.22783.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:57.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc620e25640 10 monclient: _renew_subs 2024-05-04T18:18:57.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc620e25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc620e25640 1 -- 172.21.15.19:0/978704061 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61c1d8a00 con 0x7fc61c071b20 2024-05-04T18:18:57.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.072+0000 7fc620e25640 1 librados: init done 2024-05-04T18:18:57.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.073+0000 7fc620e25640 10 client.22783.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:57.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.073+0000 7fc620e25640 15 client.22783.objecter _session_command_op_assign -1 1 2024-05-04T18:18:57.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.073+0000 7fc620e25640 20 client.22783.objecter _get_session osd=-1 returning homeless 2024-05-04T18:18:57.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.073+0000 7fc620e25640 10 client.22783.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:57.077 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.073+0000 7fc620e25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.078 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.073+0000 7fc620e25640 1 -- 172.21.15.19:0/978704061 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc61c0af910 con 0x7fc61c071b20 2024-05-04T18:18:57.092 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.091+0000 7fc612ffd640 1 -- 172.21.15.19:0/978704061 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc60401a070 con 0x7fc61c071b20 2024-05-04T18:18:57.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.092+0000 7fc612ffd640 1 --2- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc5fc06d7c0 0x7fc5fc06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.092+0000 7fc619d9b640 1 --2- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc5fc06d7c0 0x7fc5fc06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.092+0000 7fc619d9b640 10 monclient: get_auth_request con 0x7fc5fc06d7c0 auth_method 0 2024-05-04T18:18:57.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.092+0000 7fc619d9b640 1 --2- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc5fc06d7c0 0x7fc5fc06fc80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc60c036000 tx=0x7fc60c03b040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:57.093 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.092+0000 7fc612ffd640 10 client.22783.objecter ms_handle_connect 0x7fc5fc06d7c0 2024-05-04T18:18:57.099 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 1 -- 172.21.15.19:0/978704061 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7fc604056900 con 0x7fc61c071b20 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 10 client.22783.objecter ms_dispatch 0x7fc61c000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 3 client.22783.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 3 client.22783.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 10 client.22783.objecter checking command 1 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 1 --2- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] conn(0x7fc5fc075620 0x7fc5fc077ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 20 client.22783.objecter _get_session s=0x7fc5fc074f50 osd=5 3 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 20 client.22783.objecter put_session s=0x7fc5fc074f50 osd=5 3 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 15 client.22783.objecter _session_command_op_remove -1 1 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 20 client.22783.objecter _get_session s=0x7fc5fc074f50 osd=5 3 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 20 client.22783.objecter get_session s=0x7fc5fc074f50 osd=5 3 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 15 client.22783.objecter _session_command_op_assign 5 1 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 20 client.22783.objecter put_session s=0x7fc5fc074f50 osd=5 4 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 10 client.22783.objecter _send_command 1 2024-05-04T18:18:57.100 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 1 -- 172.21.15.19:0/978704061 --> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5fc078190 con 0x7fc5fc075620 2024-05-04T18:18:57.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 20 client.22783.objecter dump_active .. 0 homeless 2024-05-04T18:18:57.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 1 -- 172.21.15.19:0/978704061 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=1186) ==== 24+0+0 (secure 0 0 0) 0x7fc604054b00 con 0x7fc61c071b20 2024-05-04T18:18:57.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.098+0000 7fc612ffd640 10 monclient: handle_get_version_reply finishing 1 version 1186 2024-05-04T18:18:57.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.099+0000 7fc61ad9d640 1 --2- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] conn(0x7fc5fc075620 0x7fc5fc077ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.101 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.099+0000 7fc61ad9d640 10 monclient: get_auth_request con 0x7fc5fc075620 auth_method 0 2024-05-04T18:18:57.103 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.102+0000 7fc61ad9d640 1 --2- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] conn(0x7fc5fc075620 0x7fc5fc077ac0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:57.103 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.102+0000 7fc612ffd640 10 client.22783.objecter ms_handle_connect 0x7fc5fc075620 2024-05-04T18:18:57.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.126+0000 7fc612ffd640 1 -- 172.21.15.19:0/978704061 <== osd.5 v2:172.21.15.186:6810/3728813703 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc5fc078190 con 0x7fc5fc075620 2024-05-04T18:18:57.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.126+0000 7fc612ffd640 10 client.22783.objecter ms_dispatch 0x7fc61c000b90 command_reply(tid 1: 0 ) 2024-05-04T18:18:57.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.126+0000 7fc612ffd640 10 client.22783.objecter _finish_command 1 = system:0 2024-05-04T18:18:57.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.126+0000 7fc612ffd640 20 client.22783.objecter put_session s=0x7fc5fc074f50 osd=5 4 2024-05-04T18:18:57.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.126+0000 7fc612ffd640 15 client.22783.objecter _session_command_op_remove 5 1 2024-05-04T18:18:57.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.139+0000 7fc620e25640 10 client.22783.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-04T18:18:57.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.139+0000 7fc620e25640 15 client.22783.objecter _session_command_op_assign -1 2 2024-05-04T18:18:57.140 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.139+0000 7fc620e25640 20 client.22783.objecter _get_session s=0x7fc5fc074f50 osd=5 3 2024-05-04T18:18:57.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.139+0000 7fc620e25640 20 client.22783.objecter put_session s=0x7fc5fc074f50 osd=5 3 2024-05-04T18:18:57.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.139+0000 7fc620e25640 20 client.22783.objecter _get_session s=0x7fc5fc074f50 osd=5 3 2024-05-04T18:18:57.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.139+0000 7fc620e25640 15 client.22783.objecter _session_command_op_remove -1 2 2024-05-04T18:18:57.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.139+0000 7fc620e25640 20 client.22783.objecter get_session s=0x7fc5fc074f50 osd=5 3 2024-05-04T18:18:57.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.139+0000 7fc620e25640 15 client.22783.objecter _session_command_op_assign 5 2 2024-05-04T18:18:57.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.139+0000 7fc620e25640 20 client.22783.objecter put_session s=0x7fc5fc074f50 osd=5 4 2024-05-04T18:18:57.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.139+0000 7fc620e25640 10 client.22783.objecter _send_command 2 2024-05-04T18:18:57.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.139+0000 7fc620e25640 1 -- 172.21.15.19:0/978704061 --> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc61c068b10 con 0x7fc5fc075620 2024-05-04T18:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc612ffd640 1 -- 172.21.15.19:0/978704061 <== osd.5 v2:172.21.15.186:6810/3728813703 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc61c068b10 con 0x7fc5fc075620 2024-05-04T18:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc612ffd640 10 client.22783.objecter ms_dispatch 0x7fc61c000b90 command_reply(tid 2: 0 ) 2024-05-04T18:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc612ffd640 10 client.22783.objecter _finish_command 2 = system:0 2024-05-04T18:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc612ffd640 20 client.22783.objecter put_session s=0x7fc5fc074f50 osd=5 4 2024-05-04T18:18:57.154 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc612ffd640 15 client.22783.objecter _session_command_op_remove 5 2 2024-05-04T18:18:57.155 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-05-04T18:18:57.155 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T18:18:57.155 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-05-04T18:18:57.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 10 librados: watch_flush enter 2024-05-04T18:18:57.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 10 librados: watch_flush exit 2024-05-04T18:18:57.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 10 client.22783.objecter close_session for osd.5 2024-05-04T18:18:57.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 1 -- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] conn(0x7fc5fc075620 msgr2=0x7fc5fc077ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 1 --2- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:6810/3728813703,v1:172.21.15.186:6811/3728813703] conn(0x7fc5fc075620 0x7fc5fc077ac0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.156 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 20 client.22783.objecter put_session s=0x7fc5fc074f50 osd=5 1 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 20 client.22783.objecter shutdown clearing up homeless session... 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 10 client.22783.objecter successfully canceled tick 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 1 -- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc5fc06d7c0 msgr2=0x7fc5fc06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 1 --2- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc5fc06d7c0 0x7fc5fc06fc80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc60c036000 tx=0x7fc60c03b040 comp rx=0 tx=0).stop 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 10 monclient: shutdown 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 1 -- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc61c071b20 msgr2=0x7fc61c1d84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 1 --2- 172.21.15.19:0/978704061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc61c071b20 0x7fc61c1d84c0 secure :-1 s=READY pgs=5836 cs=0 l=1 rev1=1 crypto rx=0x7fc60400c3c0 tx=0x7fc60400c880 comp rx=0 tx=0).stop 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc61ad9d640 1 -- 172.21.15.19:0/978704061 reap_dead start 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 1 -- 172.21.15.19:0/978704061 shutdown_connections 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 1 -- 172.21.15.19:0/978704061 >> 172.21.15.19:0/978704061 conn(0x7fc61c0679a0 msgr2=0x7fc61c0b5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 1 -- 172.21.15.19:0/978704061 shutdown_connections 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 1 -- 172.21.15.19:0/978704061 wait complete. 2024-05-04T18:18:57.157 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.153+0000 7fc620e25640 1 librados: shutdown 2024-05-04T18:18:57.165 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-04T18:18:57.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.287+0000 7f70128d9640 10 monclient: get_monmap_and_config 2024-05-04T18:18:57.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.287+0000 7f70128d9640 10 monclient: build_initial_monmap 2024-05-04T18:18:57.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.287+0000 7f70128d9640 10 monclient: monmap: 2024-05-04T18:18:57.288 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:57.288 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.288 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:57.288866+0000 2024-05-04T18:18:57.289 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:57.288866+0000 2024-05-04T18:18:57.289 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:57.289 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.289 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:57.289 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:57.289 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:57.289 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:57.289 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:57.289 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.287+0000 7f70128d9640 1 Processor -- start 2024-05-04T18:18:57.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.287+0000 7f70128d9640 1 -- start start 2024-05-04T18:18:57.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.287+0000 7f70128d9640 10 monclient: init 2024-05-04T18:18:57.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:57.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-05-04T18:18:57.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f700c0a9720 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:57.291 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f700c0a9b00 0x7f700c0aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 10 monclient(hunting): picked mon.noname-c con 0x7f700c0a9b00 addr v2:172.21.15.186:3300/0 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f700c0b2e90 0x7f700c13fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 10 monclient(hunting): picked mon.noname-a con 0x7f700c0b2e90 addr v2:172.21.15.19:3300/0 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f700c0abc40 con 0x7f700c0a9720 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f700c146fa0 con 0x7f700c0b2e90 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f700c146090 con 0x7f700c0a9b00 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.288+0000 7f70128d9640 10 monclient(hunting): authenticate will time out at 3359.403169s 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.289+0000 7f7010e76640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f700c0b2e90 0x7f700c13fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.289+0000 7f700b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f700c0a9b00 0x7f700c0aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.289+0000 7f700b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f700c0a9b00 0x7f700c0aa890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:45444/0 (socket says 172.21.15.19:45444) 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.289+0000 7f700b7fe640 1 -- 172.21.15.19:0/94571645 learned_addr learned my addr 172.21.15.19:0/94571645 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.289+0000 7f7010e76640 10 monclient(hunting): get_auth_request con 0x7f700c0b2e90 auth_method 0 2024-05-04T18:18:57.292 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.289+0000 7f7010e76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.289+0000 7f7010e76640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.289+0000 7f7010e76640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.289+0000 7f700b7fe640 10 monclient(hunting): get_auth_request con 0x7f700c0a9b00 auth_method 0 2024-05-04T18:18:57.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.289+0000 7f700b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.289+0000 7f700b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.289+0000 7f700b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.290+0000 7f700b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.290+0000 7f700b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.293 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.290+0000 7f700b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.290+0000 7f7010e76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.290+0000 7f7010e76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.290+0000 7f7010e76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.292+0000 7f700b7fe640 10 monclient(hunting): handle_auth_done global_id 22786 payload 274 2024-05-04T18:18:57.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.292+0000 7f700b7fe640 1 -- 172.21.15.19:0/94571645 >> v2:172.21.15.19:3300/0 conn(0x7f700c0b2e90 msgr2=0x7f700c13fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:57.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.292+0000 7f700b7fe640 1 --2- 172.21.15.19:0/94571645 >> v2:172.21.15.19:3300/0 conn(0x7f700c0b2e90 0x7f700c13fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.292+0000 7f700b7fe640 1 -- 172.21.15.19:0/94571645 >> v1:172.21.15.19:6789/0 conn(0x7f700c0a9720 legacy=0x7f700c146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.292+0000 7f700b7fe640 10 monclient: _finish_hunting 0 2024-05-04T18:18:57.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.292+0000 7f700b7fe640 1 monclient: found mon.noname-c 2024-05-04T18:18:57.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.292+0000 7f700b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:57.294 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.292+0000 7f700b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.292+0000 7f700b7fe640 1 -- 172.21.15.19:0/94571645 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f700c146370 con 0x7f700c0a9b00 2024-05-04T18:18:57.295 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.294+0000 7f700b7fe640 1 --2- 172.21.15.19:0/94571645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f700c0a9b00 0x7f700c0aa890 secure :-1 s=READY pgs=5837 cs=0 l=1 rev1=1 crypto rx=0x7f7000009870 tx=0x7f700002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99c1b0eb0f87ce30 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:57.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 1 -- 172.21.15.19:0/94571645 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7000034070 con 0x7f700c0a9b00 2024-05-04T18:18:57.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:57.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T18:18:57.299 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 10 monclient: dump: 2024-05-04T18:18:57.299 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:57.299 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.299 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.299 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.299 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 10 monclient: _finish_auth 0 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 10 monclient: _check_auth_tickets 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 1 -- 172.21.15.19:0/94571645 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7000030860 con 0x7f700c0a9b00 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f70128d9640 5 monclient: authenticate success, global_id 22786 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f70128d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f70128d9640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 1 -- 172.21.15.19:0/94571645 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7000030c30 con 0x7f700c0a9b00 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:57.300 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f70128d9640 4 set_mon_vals no callback set 2024-05-04T18:18:57.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:57.301 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.297+0000 7f700affd640 10 monclient: dump: 2024-05-04T18:18:57.301 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:57.301 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.301 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.301 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.301 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:57.301 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.301 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:57.301 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:57.301 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:57.301 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.298+0000 7f70128d9640 10 monclient: shutdown 2024-05-04T18:18:57.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.298+0000 7f70128d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:57.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.298+0000 7f70128d9640 1 -- 172.21.15.19:0/94571645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f700c0a9b00 msgr2=0x7f700c0aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.298+0000 7f70128d9640 1 --2- 172.21.15.19:0/94571645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f700c0a9b00 0x7f700c0aa890 secure :-1 s=READY pgs=5837 cs=0 l=1 rev1=1 crypto rx=0x7f7000009870 tx=0x7f700002b9a0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.304 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.298+0000 7f70128d9640 1 -- 172.21.15.19:0/94571645 shutdown_connections 2024-05-04T18:18:57.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.298+0000 7f70128d9640 1 --2- 172.21.15.19:0/94571645 >> v2:172.21.15.19:3300/0 conn(0x7f700c0b2e90 0x7f700c13fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.298+0000 7f70128d9640 1 --2- 172.21.15.19:0/94571645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f700c0a9b00 0x7f700c0aa890 unknown :-1 s=CLOSED pgs=5837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.298+0000 7f70128d9640 1 -- 172.21.15.19:0/94571645 >> 172.21.15.19:0/94571645 conn(0x7f700c067780 msgr2=0x7f700c0b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:57.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.298+0000 7f70128d9640 1 -- 172.21.15.19:0/94571645 shutdown_connections 2024-05-04T18:18:57.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.298+0000 7f70128d9640 1 -- 172.21.15.19:0/94571645 wait complete. 2024-05-04T18:18:57.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.299+0000 7f70128d9640 10 monclient: build_initial_monmap 2024-05-04T18:18:57.305 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.299+0000 7f70128d9640 10 monclient: monmap: 2024-05-04T18:18:57.305 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:57.305 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:57.305 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:57.305 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:57.305 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.299+0000 7f70128d9640 1 librados: starting msgr at 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.299+0000 7f70128d9640 1 librados: starting objecter 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.299+0000 7f70128d9640 1 Processor -- start 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.299+0000 7f70128d9640 1 -- start start 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.299+0000 7f70128d9640 1 librados: setting wanted keys 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.299+0000 7f70128d9640 1 librados: calling monclient init 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.299+0000 7f70128d9640 10 monclient: init 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:57.306 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f700c0a9720 0x7f700c1455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 10 monclient(hunting): picked mon.noname-a con 0x7f700c0a9720 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f700c0a9b00 0x7f700c135e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 10 monclient(hunting): picked mon.noname-c con 0x7f700c0a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f700c0b2e90 0x7f700c1d7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 10 monclient(hunting): picked mon.noname-b con 0x7f700c0b2e90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f700c140770 con 0x7f700c0a9720 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f700c1405f0 con 0x7f700c0b2e90 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f700c14d420 con 0x7f700c0a9b00 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.300+0000 7f70128d9640 10 monclient(hunting): authenticate will time out at 3359.415176s 2024-05-04T18:18:57.307 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700b7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f700c0a9b00 0x7f700c135e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f7010e76640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f700c0b2e90 0x7f700c1d7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700bfff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f700c0a9720 0x7f700c1455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700b7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f700c0a9b00 0x7f700c135e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58238/0 (socket says 172.21.15.19:58238) 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700b7fe640 1 -- 172.21.15.19:0/3248525784 learned_addr learned my addr 172.21.15.19:0/3248525784 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700b7fe640 10 monclient(hunting): get_auth_request con 0x7f700c0a9b00 auth_method 0 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f7010e76640 10 monclient(hunting): get_auth_request con 0x7f700c0b2e90 auth_method 0 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f7010e76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f7010e76640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f7010e76640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700bfff640 10 monclient(hunting): get_auth_request con 0x7f700c0a9720 auth_method 0 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700bfff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.308 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.301+0000 7f700b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.302+0000 7f700b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.302+0000 7f700b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.302+0000 7f7010e76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.302+0000 7f7010e76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.302+0000 7f7010e76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.302+0000 7f700bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.302+0000 7f700bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.309 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.302+0000 7f700bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.304+0000 7f7010e76640 10 monclient(hunting): handle_auth_done global_id 22789 payload 824 2024-05-04T18:18:57.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.304+0000 7f7010e76640 1 -- 172.21.15.19:0/3248525784 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f700c0a9b00 msgr2=0x7f700c135e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.304+0000 7f7010e76640 1 --2- 172.21.15.19:0/3248525784 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f700c0a9b00 0x7f700c135e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.304+0000 7f7010e76640 1 -- 172.21.15.19:0/3248525784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f700c0a9720 msgr2=0x7f700c1455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.304+0000 7f7010e76640 1 --2- 172.21.15.19:0/3248525784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f700c0a9720 0x7f700c1455a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.310 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.304+0000 7f7010e76640 10 monclient: _finish_hunting 0 2024-05-04T18:18:57.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.304+0000 7f7010e76640 1 monclient: found mon.noname-b 2024-05-04T18:18:57.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.304+0000 7f7010e76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:57.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.304+0000 7f7010e76640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.305+0000 7f7010e76640 1 -- 172.21.15.19:0/3248525784 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f700c144bf0 con 0x7f700c0b2e90 2024-05-04T18:18:57.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.307+0000 7f7010e76640 1 --2- 172.21.15.19:0/3248525784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f700c0b2e90 0x7f700c1d7d00 secure :-1 s=READY pgs=5838 cs=0 l=1 rev1=1 crypto rx=0x7f700400eba0 tx=0x7f700400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:57.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.307+0000 7f7008ff9640 10 client.?.objecter ms_handle_connect 0x7f700c0b2e90 2024-05-04T18:18:57.311 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.307+0000 7f7008ff9640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 1 -- 172.21.15.19:0/3248525784 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7004018020 con 0x7f700c0b2e90 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 10 monclient: dump: 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.313 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 10 monclient: _finish_auth 0 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 10 monclient: _check_auth_tickets 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 1 -- 172.21.15.19:0/3248525784 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f700400cd20 con 0x7f700c0b2e90 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 1 -- 172.21.15.19:0/3248525784 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70040146c0 con 0x7f700c0b2e90 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f7008ff9640 10 monclient: dump: 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:57.314 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.315 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:57.315 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:57.315 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:57.315 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f70128d9640 5 monclient: authenticate success, global_id 22789 2024-05-04T18:18:57.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f70128d9640 10 monclient: _renew_subs 2024-05-04T18:18:57.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f70128d9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.315 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f70128d9640 1 -- 172.21.15.19:0/3248525784 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f700c13ef30 con 0x7f700c0b2e90 2024-05-04T18:18:57.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f700a7fc640 4 set_mon_vals no callback set 2024-05-04T18:18:57.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f70128d9640 10 client.22789.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:57.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f70128d9640 10 monclient: _renew_subs 2024-05-04T18:18:57.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f70128d9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f70128d9640 1 -- 172.21.15.19:0/3248525784 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f700c145ae0 con 0x7f700c0b2e90 2024-05-04T18:18:57.316 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.311+0000 7f70128d9640 1 librados: init done 2024-05-04T18:18:57.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.312+0000 7f70128d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:57.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.312+0000 7f70128d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:57.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.312+0000 7f70128d9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.317 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.312+0000 7f70128d9640 1 -- 172.21.15.19:0/3248525784 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f700c06a700 con 0x7f700c0b2e90 2024-05-04T18:18:57.333 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.332+0000 7f7008ff9640 1 -- 172.21.15.19:0/3248525784 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7004005d10 con 0x7f700c0b2e90 2024-05-04T18:18:57.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.332+0000 7f7008ff9640 1 --2- 172.21.15.19:0/3248525784 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f6ff806d810 0x7f6ff806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.332+0000 7f700bfff640 1 --2- 172.21.15.19:0/3248525784 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f6ff806d810 0x7f6ff806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.332+0000 7f700bfff640 10 monclient: get_auth_request con 0x7f6ff806d810 auth_method 0 2024-05-04T18:18:57.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.333+0000 7f700bfff640 1 --2- 172.21.15.19:0/3248525784 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f6ff806d810 0x7f6ff806fcd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc001780 tx=0x7f6ffc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:57.334 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.333+0000 7f7008ff9640 10 client.22789.objecter ms_handle_connect 0x7f6ff806d810 2024-05-04T18:18:57.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.568+0000 7f7008ff9640 1 -- 172.21.15.19:0/3248525784 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f7004058380 con 0x7f700c0b2e90 2024-05-04T18:18:57.568 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.568+0000 7f7008ff9640 10 client.22789.objecter ms_dispatch 0x7f700c000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:57.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.568+0000 7f7008ff9640 3 client.22789.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:57.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.568+0000 7f7008ff9640 3 client.22789.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:57.569 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.568+0000 7f7008ff9640 20 client.22789.objecter dump_active .. 0 homeless 2024-05-04T18:18:57.578 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.577+0000 7f7008ff9640 1 -- 172.21.15.19:0/3248525784 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7004022030 con 0x7f700c0b2e90 2024-05-04T18:18:57.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.577+0000 7f7008ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:57.579 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.577+0000 7f7008ff9640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:18:57.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.782+0000 7f70128d9640 1 -- 172.21.15.19:0/3248525784 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f700c073120 con 0x7f6ff806d810 2024-05-04T18:18:57.783 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.782+0000 7f7008ff9640 1 -- 172.21.15.19:0/3248525784 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f700c073120 con 0x7f6ff806d810 2024-05-04T18:18:57.784 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 5 to deep-scrub 2024-05-04T18:18:57.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 10 librados: watch_flush enter 2024-05-04T18:18:57.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 10 librados: watch_flush exit 2024-05-04T18:18:57.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 20 client.22789.objecter shutdown clearing up homeless session... 2024-05-04T18:18:57.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 10 client.22789.objecter successfully canceled tick 2024-05-04T18:18:57.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 1 -- 172.21.15.19:0/3248525784 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f6ff806d810 msgr2=0x7f6ff806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 1 --2- 172.21.15.19:0/3248525784 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f6ff806d810 0x7f6ff806fcd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc001780 tx=0x7f6ffc009040 comp rx=0 tx=0).stop 2024-05-04T18:18:57.786 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 10 monclient: shutdown 2024-05-04T18:18:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 1 -- 172.21.15.19:0/3248525784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f700c0b2e90 msgr2=0x7f700c1d7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 1 --2- 172.21.15.19:0/3248525784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f700c0b2e90 0x7f700c1d7d00 secure :-1 s=READY pgs=5838 cs=0 l=1 rev1=1 crypto rx=0x7f700400eba0 tx=0x7f700400c490 comp rx=0 tx=0).stop 2024-05-04T18:18:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 1 -- 172.21.15.19:0/3248525784 shutdown_connections 2024-05-04T18:18:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 1 --2- 172.21.15.19:0/3248525784 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f6ff806d810 0x7f6ff806fcd0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 1 --2- 172.21.15.19:0/3248525784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f700c0b2e90 0x7f700c1d7d00 unknown :-1 s=CLOSED pgs=5838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 1 --2- 172.21.15.19:0/3248525784 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f700c0a9b00 0x7f700c135e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 1 --2- 172.21.15.19:0/3248525784 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f700c0a9720 0x7f700c1455a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 1 -- 172.21.15.19:0/3248525784 >> 172.21.15.19:0/3248525784 conn(0x7f700c067780 msgr2=0x7f700c0b3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 1 -- 172.21.15.19:0/3248525784 shutdown_connections 2024-05-04T18:18:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.784+0000 7f70128d9640 1 -- 172.21.15.19:0/3248525784 wait complete. 2024-05-04T18:18:57.787 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.785+0000 7f70128d9640 1 librados: shutdown 2024-05-04T18:18:57.799 INFO:tasks.ceph:Scrubbing osd.6 2024-05-04T18:18:57.799 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-04T18:18:57.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.907+0000 7f245f377640 10 monclient: get_monmap_and_config 2024-05-04T18:18:57.908 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.907+0000 7f245f377640 10 monclient: build_initial_monmap 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.907+0000 7f245f377640 10 monclient: monmap: 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:57.909218+0000 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:57.909218+0000 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:57.909 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.910 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.907+0000 7f245f377640 1 Processor -- start 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 1 -- start start 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 10 monclient: init 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f24580a9720 0x7f245813fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 10 monclient(hunting): picked mon.noname-c con 0x7f24580a9720 addr v2:172.21.15.186:3300/0 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24580a9b00 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2458140370 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24580ab350 con 0x7f24580a9b00 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24581407a0 con 0x7f2458140370 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f24580ab630 con 0x7f24580a9720 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:57.911 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.908+0000 7f245f377640 10 monclient(hunting): authenticate will time out at 3360.023495s 2024-05-04T18:18:57.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.909+0000 7f245c912640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f24580a9b00 0x7f2458142dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:53142/0 (socket says 172.21.15.19:53142) 2024-05-04T18:18:57.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.909+0000 7f245c912640 1 -- 172.21.15.19:0/682603074 learned_addr learned my addr 172.21.15.19:0/682603074 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T18:18:57.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.909+0000 7f245d113640 1 --2- 172.21.15.19:0/682603074 >> v2:172.21.15.186:3300/0 conn(0x7f24580a9720 0x7f245813fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.910+0000 7f245d113640 10 monclient(hunting): get_auth_request con 0x7f24580a9720 auth_method 0 2024-05-04T18:18:57.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.910+0000 7f245d113640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.912 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.910+0000 7f245d113640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.910+0000 7f245d113640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.910+0000 7f245d113640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.910+0000 7f245d113640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.910+0000 7f245d113640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f244ffff640 1 -- 172.21.15.19:0/682603074 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2994291224 0 0) 0x7f2448001110 con 0x7f24580a9b00 2024-05-04T18:18:57.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f244ffff640 10 monclient(hunting): _negotiate 2024-05-04T18:18:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f244ffff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f244ffff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f244ffff640 10 monclient(hunting): my global_id is 15828 2024-05-04T18:18:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f244ffff640 1 -- 172.21.15.19:0/682603074 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2444003520 con 0x7f24580a9b00 2024-05-04T18:18:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f245d113640 10 monclient(hunting): handle_auth_done global_id 22792 payload 274 2024-05-04T18:18:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f245d113640 1 -- 172.21.15.19:0/682603074 >> v1:172.21.15.186:6789/0 conn(0x7f2458140370 legacy=0x7f24581434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f245d113640 1 -- 172.21.15.19:0/682603074 >> v1:172.21.15.19:6789/0 conn(0x7f24580a9b00 legacy=0x7f2458142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f245d113640 10 monclient: _finish_hunting 0 2024-05-04T18:18:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f245d113640 1 monclient: found mon.noname-c 2024-05-04T18:18:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f245d113640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f245d113640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.912+0000 7f245d113640 1 -- 172.21.15.19:0/682603074 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24580ab910 con 0x7f24580a9720 2024-05-04T18:18:57.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.915+0000 7f245d113640 1 --2- 172.21.15.19:0/682603074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24580a9720 0x7f245813fe30 secure :-1 s=READY pgs=5839 cs=0 l=1 rev1=1 crypto rx=0x7f24540097a0 tx=0x7f245402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e48d341a5284835 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 1 -- 172.21.15.19:0/682603074 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24540315a0 con 0x7f24580a9720 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 10 monclient: dump: 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:57.921 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 10 monclient: _finish_auth 0 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 10 monclient: _check_auth_tickets 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 1 -- 172.21.15.19:0/682603074 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2454037070 con 0x7f24580a9720 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 1 -- 172.21.15.19:0/682603074 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454031960 con 0x7f24580a9720 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f245f377640 5 monclient: authenticate success, global_id 22792 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f245f377640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f245f377640 4 set_mon_vals no callback set 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f244ffff640 10 monclient: dump: 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:57.922 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.923 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.923 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.923 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:57.923 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.923 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:57.923 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:57.923 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:57.923 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.926 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f245f377640 10 monclient: shutdown 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f245f377640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f245f377640 1 -- 172.21.15.19:0/682603074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24580a9720 msgr2=0x7f245813fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f245f377640 1 --2- 172.21.15.19:0/682603074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24580a9720 0x7f245813fe30 secure :-1 s=READY pgs=5839 cs=0 l=1 rev1=1 crypto rx=0x7f24540097a0 tx=0x7f245402b840 comp rx=0 tx=0).stop 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f245f377640 1 -- 172.21.15.19:0/682603074 shutdown_connections 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f245f377640 1 --2- 172.21.15.19:0/682603074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24580a9720 0x7f245813fe30 unknown :-1 s=CLOSED pgs=5839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f245f377640 1 -- 172.21.15.19:0/682603074 >> 172.21.15.19:0/682603074 conn(0x7f2458067750 msgr2=0x7f24580b8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f245f377640 1 -- 172.21.15.19:0/682603074 shutdown_connections 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.920+0000 7f245f377640 1 -- 172.21.15.19:0/682603074 wait complete. 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.921+0000 7f245f377640 10 monclient: build_initial_monmap 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.921+0000 7f245f377640 10 monclient: monmap: 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:57.927 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.921+0000 7f245f377640 1 librados: starting msgr at 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.921+0000 7f245f377640 1 librados: starting objecter 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.921+0000 7f245f377640 1 Processor -- start 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.921+0000 7f245f377640 1 -- start start 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.921+0000 7f245f377640 1 librados: setting wanted keys 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.921+0000 7f245f377640 1 librados: calling monclient init 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.921+0000 7f245f377640 10 monclient: init 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 10 monclient: _add_conns ranks=[2,1,0] 2024-05-04T18:18:57.928 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f24580a9720 0x7f245813abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 10 monclient(hunting): picked mon.noname-c con 0x7f24580a9720 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24580a9b00 0x7f24580b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 10 monclient(hunting): picked mon.noname-b con 0x7f24580a9b00 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2458140370 0x7f2458137af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 10 monclient(hunting): picked mon.noname-a con 0x7f2458140370 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f245814e580 con 0x7f2458140370 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f245814e400 con 0x7f24580a9b00 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f245814e700 con 0x7f24580a9720 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245f377640 10 monclient(hunting): authenticate will time out at 3360.036858s 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245d914640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2458140370 0x7f2458137af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245d113640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f24580a9720 0x7f245813abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.929 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.922+0000 7f245c912640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24580a9b00 0x7f24580b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d113640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f24580a9720 0x7f245813abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58252/0 (socket says 172.21.15.19:58252) 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d113640 1 -- 172.21.15.19:0/273714357 learned_addr learned my addr 172.21.15.19:0/273714357 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d113640 10 monclient(hunting): get_auth_request con 0x7f24580a9720 auth_method 0 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d113640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d113640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d113640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d914640 10 monclient(hunting): get_auth_request con 0x7f2458140370 auth_method 0 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d914640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d914640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d914640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245c912640 10 monclient(hunting): get_auth_request con 0x7f24580a9b00 auth_method 0 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245c912640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245c912640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245c912640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d113640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.930 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d113640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d113640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245c912640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245c912640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245c912640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d914640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:57.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d914640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:57.931 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.923+0000 7f245d914640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.926+0000 7f245c912640 10 monclient(hunting): handle_auth_done global_id 22795 payload 824 2024-05-04T18:18:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.926+0000 7f245c912640 1 -- 172.21.15.19:0/273714357 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f24580a9720 msgr2=0x7f245813abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.926+0000 7f245c912640 1 --2- 172.21.15.19:0/273714357 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f24580a9720 0x7f245813abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.926+0000 7f245c912640 1 -- 172.21.15.19:0/273714357 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2458140370 msgr2=0x7f2458137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.926+0000 7f245c912640 1 --2- 172.21.15.19:0/273714357 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f2458140370 0x7f2458137af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.926+0000 7f245c912640 10 monclient: _finish_hunting 0 2024-05-04T18:18:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.926+0000 7f245c912640 1 monclient: found mon.noname-b 2024-05-04T18:18:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.926+0000 7f245c912640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.926+0000 7f245c912640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.926+0000 7f245c912640 1 -- 172.21.15.19:0/273714357 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f245813c1c0 con 0x7f24580a9b00 2024-05-04T18:18:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.928+0000 7f245c912640 1 --2- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24580a9b00 0x7f24580b18e0 secure :-1 s=READY pgs=5840 cs=0 l=1 rev1=1 crypto rx=0x7f244800c3c0 tx=0x7f244800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:57.932 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.928+0000 7f244dffb640 10 client.?.objecter ms_handle_connect 0x7f24580a9b00 2024-05-04T18:18:57.933 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.928+0000 7f244dffb640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:57.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 1 -- 172.21.15.19:0/273714357 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2448017070 con 0x7f24580a9b00 2024-05-04T18:18:57.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:57.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:57.934 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 10 monclient: dump: 2024-05-04T18:18:57.934 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:57.934 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.934 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.934 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.934 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:57.934 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.935 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:57.935 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:57.935 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:57.935 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 10 monclient: _finish_auth 0 2024-05-04T18:18:57.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 10 monclient: _check_auth_tickets 2024-05-04T18:18:57.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:57.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 1 -- 172.21.15.19:0/273714357 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2448005ce0 con 0x7f24580a9b00 2024-05-04T18:18:57.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:57.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 1 -- 172.21.15.19:0/273714357 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24480072f0 con 0x7f24580a9b00 2024-05-04T18:18:57.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f245f377640 5 monclient: authenticate success, global_id 22795 2024-05-04T18:18:57.935 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244dffb640 10 monclient: dump: 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f245f377640 10 monclient: _renew_subs 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f245f377640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f245f377640 1 -- 172.21.15.19:0/273714357 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f245813b530 con 0x7f24580a9b00 2024-05-04T18:18:57.936 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f244f7fe640 4 set_mon_vals no callback set 2024-05-04T18:18:57.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f245f377640 10 client.22795.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:57.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f245f377640 10 monclient: _renew_subs 2024-05-04T18:18:57.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f245f377640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f245f377640 1 -- 172.21.15.19:0/273714357 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f245813b100 con 0x7f24580a9b00 2024-05-04T18:18:57.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.932+0000 7f245f377640 1 librados: init done 2024-05-04T18:18:57.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.933+0000 7f245f377640 10 client.22795.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:57.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.933+0000 7f245f377640 15 client.22795.objecter _session_command_op_assign -1 1 2024-05-04T18:18:57.937 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.933+0000 7f245f377640 20 client.22795.objecter _get_session osd=-1 returning homeless 2024-05-04T18:18:57.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.933+0000 7f245f377640 10 client.22795.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:57.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.933+0000 7f245f377640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:57.938 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.933+0000 7f245f377640 1 -- 172.21.15.19:0/273714357 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2458068b10 con 0x7f24580a9b00 2024-05-04T18:18:57.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.951+0000 7f244dffb640 1 -- 172.21.15.19:0/273714357 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f244801a070 con 0x7f24580a9b00 2024-05-04T18:18:57.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.951+0000 7f244dffb640 1 --2- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f243c06d810 0x7f243c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.952+0000 7f245d113640 1 --2- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f243c06d810 0x7f243c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.952+0000 7f245d113640 10 monclient: get_auth_request con 0x7f243c06d810 auth_method 0 2024-05-04T18:18:57.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.952+0000 7f244dffb640 10 client.22795.objecter ms_handle_connect 0x7f243c06d810 2024-05-04T18:18:57.954 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.952+0000 7f245d113640 1 --2- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f243c06d810 0x7f243c06fcd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f245402b880 tx=0x7f245403d040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:57.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 1 -- 172.21.15.19:0/273714357 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f2448056cf0 con 0x7f24580a9b00 2024-05-04T18:18:57.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 10 client.22795.objecter ms_dispatch 0x7f2458000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:57.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 3 client.22795.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:57.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 3 client.22795.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:57.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 10 client.22795.objecter checking command 1 2024-05-04T18:18:57.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 1 --2- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] conn(0x7f243c075670 0x7f243c077b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:57.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 20 client.22795.objecter _get_session s=0x7f243c074fa0 osd=6 3 2024-05-04T18:18:57.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 20 client.22795.objecter put_session s=0x7f243c074fa0 osd=6 3 2024-05-04T18:18:57.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 15 client.22795.objecter _session_command_op_remove -1 1 2024-05-04T18:18:57.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 20 client.22795.objecter _get_session s=0x7f243c074fa0 osd=6 3 2024-05-04T18:18:57.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 20 client.22795.objecter get_session s=0x7f243c074fa0 osd=6 3 2024-05-04T18:18:57.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 15 client.22795.objecter _session_command_op_assign 6 1 2024-05-04T18:18:57.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 20 client.22795.objecter put_session s=0x7f243c074fa0 osd=6 4 2024-05-04T18:18:57.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 10 client.22795.objecter _send_command 1 2024-05-04T18:18:57.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 1 -- 172.21.15.19:0/273714357 --> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f243c0781e0 con 0x7f243c075670 2024-05-04T18:18:57.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.954+0000 7f244dffb640 20 client.22795.objecter dump_active .. 0 homeless 2024-05-04T18:18:57.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.955+0000 7f245d914640 1 --2- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] conn(0x7f243c075670 0x7f243c077b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:57.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.956+0000 7f245d914640 10 monclient: get_auth_request con 0x7f243c075670 auth_method 0 2024-05-04T18:18:57.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.956+0000 7f244dffb640 1 -- 172.21.15.19:0/273714357 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=1186) ==== 24+0+0 (secure 0 0 0) 0x7f2448054ef0 con 0x7f24580a9b00 2024-05-04T18:18:57.958 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.956+0000 7f244dffb640 10 monclient: handle_get_version_reply finishing 1 version 1186 2024-05-04T18:18:57.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.958+0000 7f244dffb640 10 client.22795.objecter ms_handle_connect 0x7f243c075670 2024-05-04T18:18:57.960 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.958+0000 7f245d914640 1 --2- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] conn(0x7f243c075670 0x7f243c077b10 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:57.983 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.982+0000 7f244dffb640 1 -- 172.21.15.19:0/273714357 <== osd.6 v2:172.21.15.186:6802/4252462313 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f243c0781e0 con 0x7f243c075670 2024-05-04T18:18:57.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.982+0000 7f244dffb640 10 client.22795.objecter ms_dispatch 0x7f2458000b90 command_reply(tid 1: 0 ) 2024-05-04T18:18:57.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.982+0000 7f244dffb640 10 client.22795.objecter _finish_command 1 = system:0 2024-05-04T18:18:57.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.982+0000 7f244dffb640 20 client.22795.objecter put_session s=0x7f243c074fa0 osd=6 4 2024-05-04T18:18:57.984 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.982+0000 7f244dffb640 15 client.22795.objecter _session_command_op_remove 6 1 2024-05-04T18:18:57.996 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.995+0000 7f245f377640 10 client.22795.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-04T18:18:57.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.995+0000 7f245f377640 15 client.22795.objecter _session_command_op_assign -1 2 2024-05-04T18:18:57.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.995+0000 7f245f377640 20 client.22795.objecter _get_session s=0x7f243c074fa0 osd=6 3 2024-05-04T18:18:57.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.995+0000 7f245f377640 20 client.22795.objecter put_session s=0x7f243c074fa0 osd=6 3 2024-05-04T18:18:57.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.995+0000 7f245f377640 20 client.22795.objecter _get_session s=0x7f243c074fa0 osd=6 3 2024-05-04T18:18:57.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.995+0000 7f245f377640 15 client.22795.objecter _session_command_op_remove -1 2 2024-05-04T18:18:57.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.995+0000 7f245f377640 20 client.22795.objecter get_session s=0x7f243c074fa0 osd=6 3 2024-05-04T18:18:57.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.995+0000 7f245f377640 15 client.22795.objecter _session_command_op_assign 6 2 2024-05-04T18:18:57.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.995+0000 7f245f377640 20 client.22795.objecter put_session s=0x7f243c074fa0 osd=6 4 2024-05-04T18:18:57.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.995+0000 7f245f377640 10 client.22795.objecter _send_command 2 2024-05-04T18:18:57.997 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:57.995+0000 7f245f377640 1 -- 172.21.15.19:0/273714357 --> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f245813a9e0 con 0x7f243c075670 2024-05-04T18:18:58.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.010+0000 7f244dffb640 1 -- 172.21.15.19:0/273714357 <== osd.6 v2:172.21.15.186:6802/4252462313 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f245813a9e0 con 0x7f243c075670 2024-05-04T18:18:58.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.010+0000 7f244dffb640 10 client.22795.objecter ms_dispatch 0x7f2458000b90 command_reply(tid 2: 0 ) 2024-05-04T18:18:58.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.010+0000 7f244dffb640 10 client.22795.objecter _finish_command 2 = system:0 2024-05-04T18:18:58.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.010+0000 7f244dffb640 20 client.22795.objecter put_session s=0x7f243c074fa0 osd=6 4 2024-05-04T18:18:58.012 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.010+0000 7f244dffb640 15 client.22795.objecter _session_command_op_remove 6 2 2024-05-04T18:18:58.012 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-05-04T18:18:58.012 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T18:18:58.012 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-05-04T18:18:58.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 10 librados: watch_flush enter 2024-05-04T18:18:58.013 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 10 librados: watch_flush exit 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 10 client.22795.objecter close_session for osd.6 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 1 -- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] conn(0x7f243c075670 msgr2=0x7f243c077b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 1 --2- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:6802/4252462313,v1:172.21.15.186:6803/4252462313] conn(0x7f243c075670 0x7f243c077b10 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 20 client.22795.objecter put_session s=0x7f243c074fa0 osd=6 1 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 20 client.22795.objecter shutdown clearing up homeless session... 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 10 client.22795.objecter successfully canceled tick 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 1 -- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f243c06d810 msgr2=0x7f243c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 1 --2- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f243c06d810 0x7f243c06fcd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f245402b880 tx=0x7f245403d040 comp rx=0 tx=0).stop 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 10 monclient: shutdown 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 1 -- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24580a9b00 msgr2=0x7f24580b18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 1 --2- 172.21.15.19:0/273714357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24580a9b00 0x7f24580b18e0 secure :-1 s=READY pgs=5840 cs=0 l=1 rev1=1 crypto rx=0x7f244800c3c0 tx=0x7f244800c880 comp rx=0 tx=0).stop 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245d914640 1 -- 172.21.15.19:0/273714357 reap_dead start 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 1 -- 172.21.15.19:0/273714357 shutdown_connections 2024-05-04T18:18:58.014 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 1 -- 172.21.15.19:0/273714357 >> 172.21.15.19:0/273714357 conn(0x7f2458067750 msgr2=0x7f24580b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:58.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 1 -- 172.21.15.19:0/273714357 shutdown_connections 2024-05-04T18:18:58.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 1 -- 172.21.15.19:0/273714357 wait complete. 2024-05-04T18:18:58.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.011+0000 7f245f377640 1 librados: shutdown 2024-05-04T18:18:58.023 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-04T18:18:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.123+0000 7fc5b3198640 10 monclient: get_monmap_and_config 2024-05-04T18:18:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.123+0000 7fc5b3198640 10 monclient: build_initial_monmap 2024-05-04T18:18:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.123+0000 7fc5b3198640 10 monclient: monmap: 2024-05-04T18:18:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:58.125528+0000 2024-05-04T18:18:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:58.125528+0000 2024-05-04T18:18:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:58.125 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:58.126 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:58.126 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:58.126 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:58.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.124+0000 7fc5b3198640 1 Processor -- start 2024-05-04T18:18:58.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.124+0000 7fc5b3198640 1 -- start start 2024-05-04T18:18:58.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.124+0000 7fc5b3198640 10 monclient: init 2024-05-04T18:18:58.127 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b3198640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b3198640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b3198640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5ac0b2690 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b3198640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5ac13fd70 addr v1:172.21.15.19:6789/0 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b3198640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5ac142c70 0x7fc5ac1430e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b3198640 10 monclient(hunting): picked mon.noname-b con 0x7fc5ac142c70 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b3198640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5ac073680 con 0x7fc5ac13fd70 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b3198640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5ac143620 con 0x7fc5ac0b2690 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b3198640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b3198640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ac0b2a70 con 0x7fc5ac142c70 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b3198640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b3198640 10 monclient(hunting): authenticate will time out at 3360.239754s 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5b1735640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5ac142c70 0x7fc5ac1430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5abfff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fc5ac13fd70 0x7fc5ac1401b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:53146/0 (socket says 172.21.15.19:53146) 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.125+0000 7fc5abfff640 1 -- 172.21.15.19:0/1078525773 learned_addr learned my addr 172.21.15.19:0/1078525773 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T18:18:58.128 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.126+0000 7fc5b1735640 10 monclient(hunting): get_auth_request con 0x7fc5ac142c70 auth_method 0 2024-05-04T18:18:58.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.126+0000 7fc5b1735640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:58.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.126+0000 7fc5b1735640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:58.129 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.126+0000 7fc5b1735640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:58.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.126+0000 7fc5b1735640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:58.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.126+0000 7fc5b1735640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:58.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.126+0000 7fc5b1735640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:58.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5ab7fe640 1 -- 172.21.15.19:0/1078525773 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3852624731 0 0) 0x7fc5ac143620 con 0x7fc5ac0b2690 2024-05-04T18:18:58.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5ab7fe640 10 monclient(hunting): _negotiate 2024-05-04T18:18:58.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5ab7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:58.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:58.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5ab7fe640 10 monclient(hunting): my global_id is 22798 2024-05-04T18:18:58.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5ab7fe640 1 -- 172.21.15.19:0/1078525773 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc59c003520 con 0x7fc5ac0b2690 2024-05-04T18:18:58.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5b1735640 10 monclient(hunting): handle_auth_done global_id 18839 payload 274 2024-05-04T18:18:58.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5b1735640 1 -- 172.21.15.19:0/1078525773 >> v1:172.21.15.186:6789/0 conn(0x7fc5ac0b2690 legacy=0x7fc5ac0b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.130 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5b1735640 1 -- 172.21.15.19:0/1078525773 >> v1:172.21.15.19:6789/0 conn(0x7fc5ac13fd70 legacy=0x7fc5ac1401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5b1735640 10 monclient: _finish_hunting 0 2024-05-04T18:18:58.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5b1735640 1 monclient: found mon.noname-b 2024-05-04T18:18:58.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5b1735640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:58.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5b1735640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T18:18:58.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.128+0000 7fc5b1735640 1 -- 172.21.15.19:0/1078525773 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ac1437f0 con 0x7fc5ac142c70 2024-05-04T18:18:58.131 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.130+0000 7fc5b1735640 1 --2- 172.21.15.19:0/1078525773 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5ac142c70 0x7fc5ac1430e0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4009dc0 tx=0x7fc5a402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e68282f96ac8eb8 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 1 -- 172.21.15.19:0/1078525773 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a4039070 con 0x7fc5ac142c70 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 10 monclient: dump: 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:58.136 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 10 monclient: _finish_auth 0 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 10 monclient: _check_auth_tickets 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 1 -- 172.21.15.19:0/1078525773 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5a402cc40 con 0x7fc5ac142c70 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5b3198640 5 monclient: authenticate success, global_id 18839 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5b3198640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5b3198640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 1 -- 172.21.15.19:0/1078525773 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a4034440 con 0x7fc5ac142c70 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5b3198640 4 set_mon_vals no callback set 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.134+0000 7fc5ab7fe640 10 monclient: dump: 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:58.137 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:58.138 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.138 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.138 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:58.138 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:58.138 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:58.138 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:58.138 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:58.138 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:58.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.135+0000 7fc5b3198640 10 monclient: shutdown 2024-05-04T18:18:58.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.135+0000 7fc5b3198640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:58.141 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.135+0000 7fc5b3198640 1 -- 172.21.15.19:0/1078525773 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5ac142c70 msgr2=0x7fc5ac1430e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.135+0000 7fc5b3198640 1 --2- 172.21.15.19:0/1078525773 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5ac142c70 0x7fc5ac1430e0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4009dc0 tx=0x7fc5a402be90 comp rx=0 tx=0).stop 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.135+0000 7fc5b3198640 1 -- 172.21.15.19:0/1078525773 shutdown_connections 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.135+0000 7fc5b3198640 1 --2- 172.21.15.19:0/1078525773 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5ac142c70 0x7fc5ac1430e0 unknown :-1 s=CLOSED pgs=3172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.135+0000 7fc5b3198640 1 -- 172.21.15.19:0/1078525773 >> 172.21.15.19:0/1078525773 conn(0x7fc5ac0b5840 msgr2=0x7fc5ac0b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.135+0000 7fc5b3198640 1 -- 172.21.15.19:0/1078525773 shutdown_connections 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.135+0000 7fc5b3198640 1 -- 172.21.15.19:0/1078525773 wait complete. 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.135+0000 7fc5b3198640 10 monclient: build_initial_monmap 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.135+0000 7fc5b3198640 10 monclient: monmap: 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:58.142 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.136+0000 7fc5b3198640 1 librados: starting msgr at 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.136+0000 7fc5b3198640 1 librados: starting objecter 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.136+0000 7fc5b3198640 1 Processor -- start 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.136+0000 7fc5b3198640 1 -- start start 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.136+0000 7fc5b3198640 1 librados: setting wanted keys 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.136+0000 7fc5b3198640 1 librados: calling monclient init 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.136+0000 7fc5b3198640 10 monclient: init 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc5ac0b2690 0x7fc5ac0729d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 10 monclient(hunting): picked mon.noname-a con 0x7fc5ac0b2690 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:58.143 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5ac13fd70 0x7fc5ac1db850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 10 monclient(hunting): picked mon.noname-b con 0x7fc5ac13fd70 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5ac142c70 0x7fc5ac1dbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 10 monclient(hunting): picked mon.noname-c con 0x7fc5ac142c70 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac073850 con 0x7fc5ac0b2690 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac0b1a20 con 0x7fc5ac13fd70 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ac14e5b0 con 0x7fc5ac142c70 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b3198640 10 monclient(hunting): authenticate will time out at 3360.251745s 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b1735640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5ac142c70 0x7fc5ac1dbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5abfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5ac13fd70 0x7fc5ac1db850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b0f34640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc5ac0b2690 0x7fc5ac0729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:58.144 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b1735640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5ac142c70 0x7fc5ac1dbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58264/0 (socket says 172.21.15.19:58264) 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b1735640 1 -- 172.21.15.19:0/1341828872 learned_addr learned my addr 172.21.15.19:0/1341828872 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b1735640 10 monclient(hunting): get_auth_request con 0x7fc5ac142c70 auth_method 0 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b1735640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b1735640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.137+0000 7fc5b1735640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5abfff640 10 monclient(hunting): get_auth_request con 0x7fc5ac13fd70 auth_method 0 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5abfff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5abfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5b0f34640 10 monclient(hunting): get_auth_request con 0x7fc5ac0b2690 auth_method 0 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5b0f34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5b0f34640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5b0f34640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:58.145 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:58.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:58.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5b1735640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:58.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5b1735640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:58.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5b1735640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:58.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5b0f34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:58.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5b0f34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:58.146 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.138+0000 7fc5b0f34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:58.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.141+0000 7fc5abfff640 10 monclient(hunting): handle_auth_done global_id 22801 payload 824 2024-05-04T18:18:58.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.141+0000 7fc5abfff640 1 -- 172.21.15.19:0/1341828872 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5ac142c70 msgr2=0x7fc5ac1dbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.141+0000 7fc5abfff640 1 --2- 172.21.15.19:0/1341828872 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5ac142c70 0x7fc5ac1dbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.141+0000 7fc5abfff640 1 -- 172.21.15.19:0/1341828872 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc5ac0b2690 msgr2=0x7fc5ac0729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.141+0000 7fc5abfff640 1 --2- 172.21.15.19:0/1341828872 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc5ac0b2690 0x7fc5ac0729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.141+0000 7fc5abfff640 10 monclient: _finish_hunting 0 2024-05-04T18:18:58.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.141+0000 7fc5abfff640 1 monclient: found mon.noname-b 2024-05-04T18:18:58.147 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.141+0000 7fc5abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:58.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.141+0000 7fc5abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:18:58.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.141+0000 7fc5abfff640 1 -- 172.21.15.19:0/1341828872 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ac1454b0 con 0x7fc5ac13fd70 2024-05-04T18:18:58.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.143+0000 7fc5abfff640 1 --2- 172.21.15.19:0/1341828872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5ac13fd70 0x7fc5ac1db850 secure :-1 s=READY pgs=5842 cs=0 l=1 rev1=1 crypto rx=0x7fc59800c370 tx=0x7fc59800c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:58.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.143+0000 7fc5a97fa640 10 client.?.objecter ms_handle_connect 0x7fc5ac13fd70 2024-05-04T18:18:58.148 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.143+0000 7fc5a97fa640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:58.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 1 -- 172.21.15.19:0/1341828872 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598017070 con 0x7fc5ac13fd70 2024-05-04T18:18:58.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:58.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:58.149 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 10 monclient: dump: 2024-05-04T18:18:58.149 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:58.149 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:58.149 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.149 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.149 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 10 monclient: _finish_auth 0 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 10 monclient: _check_auth_tickets 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 1 -- 172.21.15.19:0/1341828872 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc598005d00 con 0x7fc5ac13fd70 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5b3198640 5 monclient: authenticate success, global_id 22801 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5b3198640 10 monclient: _renew_subs 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5b3198640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5b3198640 1 -- 172.21.15.19:0/1341828872 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5ac145790 con 0x7fc5ac13fd70 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 1 -- 172.21.15.19:0/1341828872 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598007310 con 0x7fc5ac13fd70 2024-05-04T18:18:58.150 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5a97fa640 10 monclient: dump: 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5aaffd640 4 set_mon_vals no callback set 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5b3198640 10 client.22801.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:58.151 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5b3198640 10 monclient: _renew_subs 2024-05-04T18:18:58.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5b3198640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:58.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5b3198640 1 -- 172.21.15.19:0/1341828872 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ac1448a0 con 0x7fc5ac13fd70 2024-05-04T18:18:58.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.147+0000 7fc5b3198640 1 librados: init done 2024-05-04T18:18:58.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.148+0000 7fc5b3198640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:58.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.148+0000 7fc5b3198640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:58.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.148+0000 7fc5b3198640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:18:58.152 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.148+0000 7fc5b3198640 1 -- 172.21.15.19:0/1341828872 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ac145b80 con 0x7fc5ac13fd70 2024-05-04T18:18:58.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.414+0000 7fc5a97fa640 1 -- 172.21.15.19:0/1341828872 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc59801a070 con 0x7fc5ac13fd70 2024-05-04T18:18:58.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.415+0000 7fc5a97fa640 1 --2- 172.21.15.19:0/1341828872 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc59006d810 0x7fc59006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.415+0000 7fc5b0f34640 1 --2- 172.21.15.19:0/1341828872 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc59006d810 0x7fc59006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:58.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.415+0000 7fc5b0f34640 10 monclient: get_auth_request con 0x7fc59006d810 auth_method 0 2024-05-04T18:18:58.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.416+0000 7fc5a97fa640 10 client.22801.objecter ms_handle_connect 0x7fc59006d810 2024-05-04T18:18:58.417 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.416+0000 7fc5b0f34640 1 --2- 172.21.15.19:0/1341828872 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc59006d810 0x7fc59006fcd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0007fd0 tx=0x7fc5a0009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:58.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.418+0000 7fc5a97fa640 1 -- 172.21.15.19:0/1341828872 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7fc598056970 con 0x7fc5ac13fd70 2024-05-04T18:18:58.419 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.418+0000 7fc5a97fa640 10 client.22801.objecter ms_dispatch 0x7fc5ac000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:58.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.418+0000 7fc5a97fa640 3 client.22801.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:58.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.418+0000 7fc5a97fa640 3 client.22801.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:58.420 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.418+0000 7fc5a97fa640 20 client.22801.objecter dump_active .. 0 homeless 2024-05-04T18:18:58.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.429+0000 7fc5a97fa640 1 -- 172.21.15.19:0/1341828872 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc598021020 con 0x7fc5ac13fd70 2024-05-04T18:18:58.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.429+0000 7fc5a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:58.429 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.429+0000 7fc5a97fa640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:18:58.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.621+0000 7fc5b3198640 1 -- 172.21.15.19:0/1341828872 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fc5ac145960 con 0x7fc59006d810 2024-05-04T18:18:58.623 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.622+0000 7fc5a97fa640 1 -- 172.21.15.19:0/1341828872 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc5ac145960 con 0x7fc59006d810 2024-05-04T18:18:58.624 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 6 to deep-scrub 2024-05-04T18:18:58.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 10 librados: watch_flush enter 2024-05-04T18:18:58.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 10 librados: watch_flush exit 2024-05-04T18:18:58.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 20 client.22801.objecter shutdown clearing up homeless session... 2024-05-04T18:18:58.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 10 client.22801.objecter successfully canceled tick 2024-05-04T18:18:58.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 -- 172.21.15.19:0/1341828872 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc59006d810 msgr2=0x7fc59006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 --2- 172.21.15.19:0/1341828872 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc59006d810 0x7fc59006fcd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0007fd0 tx=0x7fc5a0009040 comp rx=0 tx=0).stop 2024-05-04T18:18:58.626 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 10 monclient: shutdown 2024-05-04T18:18:58.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:58.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 -- 172.21.15.19:0/1341828872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5ac13fd70 msgr2=0x7fc5ac1db850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 --2- 172.21.15.19:0/1341828872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5ac13fd70 0x7fc5ac1db850 secure :-1 s=READY pgs=5842 cs=0 l=1 rev1=1 crypto rx=0x7fc59800c370 tx=0x7fc59800c830 comp rx=0 tx=0).stop 2024-05-04T18:18:58.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 -- 172.21.15.19:0/1341828872 shutdown_connections 2024-05-04T18:18:58.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 --2- 172.21.15.19:0/1341828872 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fc59006d810 0x7fc59006fcd0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 --2- 172.21.15.19:0/1341828872 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fc5ac142c70 0x7fc5ac1dbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 --2- 172.21.15.19:0/1341828872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5ac13fd70 0x7fc5ac1db850 unknown :-1 s=CLOSED pgs=5842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 --2- 172.21.15.19:0/1341828872 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fc5ac0b2690 0x7fc5ac0729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 -- 172.21.15.19:0/1341828872 >> 172.21.15.19:0/1341828872 conn(0x7fc5ac0b5840 msgr2=0x7fc5ac13bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:58.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 -- 172.21.15.19:0/1341828872 shutdown_connections 2024-05-04T18:18:58.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 -- 172.21.15.19:0/1341828872 wait complete. 2024-05-04T18:18:58.627 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.624+0000 7fc5b3198640 1 librados: shutdown 2024-05-04T18:18:58.637 INFO:tasks.ceph:Scrubbing osd.7 2024-05-04T18:18:58.637 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-04T18:18:58.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.746+0000 7f767aa42640 10 monclient: get_monmap_and_config 2024-05-04T18:18:58.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.746+0000 7f767aa42640 10 monclient: build_initial_monmap 2024-05-04T18:18:58.747 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.746+0000 7f767aa42640 10 monclient: monmap: 2024-05-04T18:18:58.747 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:58.747 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:58.748 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:58.747982+0000 2024-05-04T18:18:58.748 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:58.747982+0000 2024-05-04T18:18:58.748 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:58.748 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:58.748 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:58.748 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:58.748 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:58.748 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:58.748 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:58.748 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:58.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.746+0000 7f767aa42640 1 Processor -- start 2024-05-04T18:18:58.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 1 -- start start 2024-05-04T18:18:58.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 10 monclient: init 2024-05-04T18:18:58.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:58.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-05-04T18:18:58.749 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f76740a9720 0x7f76740aa890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 10 monclient(hunting): picked mon.noname-a con 0x7f76740a9720 addr v2:172.21.15.19:3300/0 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f76740a9b00 0x7f76740b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 10 monclient(hunting): picked mon.noname-b con 0x7f76740a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7674146890 0x7f767413fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 10 monclient(hunting): picked mon.noname-c con 0x7f7674146890 addr v2:172.21.15.186:3300/0 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f76740abc40 con 0x7f76740a9720 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f7674146090 con 0x7f7674146890 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f7674146370 con 0x7f76740a9b00 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:58.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.747+0000 7f767aa42640 10 monclient(hunting): authenticate will time out at 3360.862292s 2024-05-04T18:18:58.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.789+0000 7f7673fff640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f76740a9720 0x7f76740aa890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:58.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.790+0000 7f7673fff640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f76740a9720 0x7f76740aa890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54382/0 (socket says 172.21.15.19:54382) 2024-05-04T18:18:58.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.790+0000 7f7673fff640 1 -- 172.21.15.19:0/3164658576 learned_addr learned my addr 172.21.15.19:0/3164658576 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:58.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.790+0000 7f7673fff640 10 monclient(hunting): get_auth_request con 0x7f76740a9720 auth_method 0 2024-05-04T18:18:58.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.790+0000 7f7673fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:58.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.790+0000 7f7673fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:58.791 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.790+0000 7f7673fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:58.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.790+0000 7f7673fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:58.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.790+0000 7f7673fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:58.792 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.790+0000 7f7673fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:58.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.792+0000 7f7673fff640 10 monclient(hunting): handle_auth_done global_id 15840 payload 274 2024-05-04T18:18:58.793 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.792+0000 7f7673fff640 1 -- 172.21.15.19:0/3164658576 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f76740a9b00 msgr2=0x7f76740b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:58.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.792+0000 7f7673fff640 1 --2- 172.21.15.19:0/3164658576 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f76740a9b00 0x7f76740b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.792+0000 7f7673fff640 1 -- 172.21.15.19:0/3164658576 >> v2:172.21.15.186:3300/0 conn(0x7f7674146890 msgr2=0x7f767413fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:58.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.792+0000 7f7673fff640 1 --2- 172.21.15.19:0/3164658576 >> v2:172.21.15.186:3300/0 conn(0x7f7674146890 0x7f767413fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.792+0000 7f7673fff640 10 monclient: _finish_hunting 0 2024-05-04T18:18:58.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.792+0000 7f7673fff640 1 monclient: found mon.noname-a 2024-05-04T18:18:58.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.792+0000 7f7673fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:58.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.792+0000 7f7673fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T18:18:58.794 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.792+0000 7f7673fff640 1 -- 172.21.15.19:0/3164658576 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7674146d20 con 0x7f76740a9720 2024-05-04T18:18:58.795 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.794+0000 7f7673fff640 1 --2- 172.21.15.19:0/3164658576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76740a9720 0x7f76740aa890 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f767406d760 tx=0x7f766402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ddacea1590763e server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:58.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 1 -- 172.21.15.19:0/3164658576 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7664035020 con 0x7f76740a9720 2024-05-04T18:18:58.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:58.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T18:18:58.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 10 monclient: dump: 2024-05-04T18:18:58.801 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:58.801 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:58.801 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.801 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.801 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:58.801 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:58.801 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:58.801 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:58.801 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:58.801 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:58.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 10 monclient: _finish_auth 0 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 10 monclient: _check_auth_tickets 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 1 -- 172.21.15.19:0/3164658576 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7664031870 con 0x7f76740a9720 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 1 -- 172.21.15.19:0/3164658576 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7664031c40 con 0x7f76740a9720 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f767aa42640 5 monclient: authenticate success, global_id 15840 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f767aa42640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f767aa42640 4 set_mon_vals no callback set 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f7672ffd640 10 monclient: dump: 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:58.802 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.803 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.803 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:58.803 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:58.803 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:58.803 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:58.803 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:58.803 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:58.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f767aa42640 10 monclient: shutdown 2024-05-04T18:18:58.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f767aa42640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:58.806 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f767aa42640 1 -- 172.21.15.19:0/3164658576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76740a9720 msgr2=0x7f76740aa890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f767aa42640 1 --2- 172.21.15.19:0/3164658576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76740a9720 0x7f76740aa890 secure :-1 s=READY pgs=3075 cs=0 l=1 rev1=1 crypto rx=0x7f767406d760 tx=0x7f766402b910 comp rx=0 tx=0).stop 2024-05-04T18:18:58.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f767aa42640 1 -- 172.21.15.19:0/3164658576 shutdown_connections 2024-05-04T18:18:58.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f767aa42640 1 --2- 172.21.15.19:0/3164658576 >> v2:172.21.15.186:3300/0 conn(0x7f7674146890 0x7f767413fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f767aa42640 1 --2- 172.21.15.19:0/3164658576 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f76740a9b00 0x7f76740b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f767aa42640 1 --2- 172.21.15.19:0/3164658576 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76740a9720 0x7f76740aa890 unknown :-1 s=CLOSED pgs=3075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.799+0000 7f767aa42640 1 -- 172.21.15.19:0/3164658576 >> 172.21.15.19:0/3164658576 conn(0x7f7674067780 msgr2=0x7f76740b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:58.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.800+0000 7f767aa42640 1 -- 172.21.15.19:0/3164658576 shutdown_connections 2024-05-04T18:18:58.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.800+0000 7f767aa42640 1 -- 172.21.15.19:0/3164658576 wait complete. 2024-05-04T18:18:58.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.800+0000 7f767aa42640 10 monclient: build_initial_monmap 2024-05-04T18:18:58.807 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.800+0000 7f767aa42640 10 monclient: monmap: 2024-05-04T18:18:58.807 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:58.807 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:58.808 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:58.808 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:58.808 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:58.808 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:58.808 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:58.808 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:58.808 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:58.808 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:58.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.800+0000 7f767aa42640 1 librados: starting msgr at 2024-05-04T18:18:58.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.800+0000 7f767aa42640 1 librados: starting objecter 2024-05-04T18:18:58.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.801+0000 7f767aa42640 1 Processor -- start 2024-05-04T18:18:58.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.801+0000 7f767aa42640 1 -- start start 2024-05-04T18:18:58.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.801+0000 7f767aa42640 1 librados: setting wanted keys 2024-05-04T18:18:58.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.801+0000 7f767aa42640 1 librados: calling monclient init 2024-05-04T18:18:58.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.801+0000 7f767aa42640 10 monclient: init 2024-05-04T18:18:58.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.801+0000 7f767aa42640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:58.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.801+0000 7f767aa42640 10 monclient: _add_conns ranks=[0,2,1] 2024-05-04T18:18:58.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.801+0000 7f767aa42640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76740a9720 0x7f76741cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.801+0000 7f767aa42640 10 monclient(hunting): picked mon.noname-a con 0x7f76740a9720 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:58.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.801+0000 7f767aa42640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f76740a9b00 0x7f76741cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.801+0000 7f767aa42640 10 monclient(hunting): picked mon.noname-c con 0x7f76740a9b00 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:58.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.801+0000 7f767aa42640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7674146890 0x7f76741cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.802+0000 7f767aa42640 10 monclient(hunting): picked mon.noname-b con 0x7f7674146890 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:58.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.802+0000 7f767aa42640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:58.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.802+0000 7f767aa42640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f7674140770 con 0x7f76740a9720 2024-05-04T18:18:58.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.802+0000 7f767aa42640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:58.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.802+0000 7f767aa42640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f76741405f0 con 0x7f7674146890 2024-05-04T18:18:58.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.802+0000 7f767aa42640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:58.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.802+0000 7f767aa42640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f7674142b50 con 0x7f76740a9b00 2024-05-04T18:18:58.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.802+0000 7f767aa42640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:58.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.802+0000 7f767aa42640 10 monclient(hunting): authenticate will time out at 3360.916681s 2024-05-04T18:18:58.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.802+0000 7f7673fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76740a9720 0x7f76741cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:58.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.803+0000 7f7673fff640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76740a9720 0x7f76741cc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:54390/0 (socket says 172.21.15.19:54390) 2024-05-04T18:18:58.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.803+0000 7f7673fff640 1 -- 172.21.15.19:0/4055503940 learned_addr learned my addr 172.21.15.19:0/4055503940 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:58.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.803+0000 7f7673fff640 10 monclient(hunting): get_auth_request con 0x7f76740a9720 auth_method 0 2024-05-04T18:18:58.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.803+0000 7f7673fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:58.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.803+0000 7f7673fff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:58.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.803+0000 7f7673fff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:58.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.803+0000 7f7673fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.803+0000 7f7673fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.803+0000 7f7673fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.804+0000 7f76737fe640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f76740a9b00 0x7f76741cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.804+0000 7f76737fe640 10 monclient(hunting): get_auth_request con 0x7f76740a9b00 auth_method 0 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.804+0000 7f76737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.804+0000 7f76737fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.804+0000 7f76737fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.804+0000 7f7678fdf640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7674146890 0x7f76741cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.805+0000 7f7678fdf640 10 monclient(hunting): get_auth_request con 0x7f7674146890 auth_method 0 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.805+0000 7f7678fdf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.805+0000 7f7678fdf640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.805+0000 7f7678fdf640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.805+0000 7f76737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:58.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.805+0000 7f76737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:58.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.805+0000 7f76737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:58.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.805+0000 7f7678fdf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:58.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.805+0000 7f7678fdf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:58.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.805+0000 7f7678fdf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:58.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.807+0000 7f7673fff640 10 monclient(hunting): handle_auth_done global_id 15843 payload 824 2024-05-04T18:18:58.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.807+0000 7f7673fff640 1 -- 172.21.15.19:0/4055503940 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f76740a9b00 msgr2=0x7f76741cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.807+0000 7f7673fff640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f76740a9b00 0x7f76741cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.807+0000 7f7673fff640 1 -- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7674146890 msgr2=0x7f76741cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.807+0000 7f7673fff640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7674146890 0x7f76741cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.807+0000 7f7673fff640 10 monclient: _finish_hunting 0 2024-05-04T18:18:58.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.807+0000 7f7673fff640 1 monclient: found mon.noname-a 2024-05-04T18:18:58.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.807+0000 7f7673fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:58.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.807+0000 7f7673fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T18:18:58.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.807+0000 7f7673fff640 1 -- 172.21.15.19:0/4055503940 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76741ce620 con 0x7f76740a9720 2024-05-04T18:18:58.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.815+0000 7f7673fff640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76740a9720 0x7f76741cc7f0 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7f7664000c00 tx=0x7f76640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:58.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.815+0000 7f7670ff9640 10 client.?.objecter ms_handle_connect 0x7f76740a9720 2024-05-04T18:18:58.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.815+0000 7f7670ff9640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:58.821 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 1 -- 172.21.15.19:0/4055503940 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766403ad60 con 0x7f76740a9720 2024-05-04T18:18:58.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:58.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T18:18:58.822 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 10 monclient: dump: 2024-05-04T18:18:58.822 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 10 monclient: _finish_auth 0 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 10 monclient: _check_auth_tickets 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 1 -- 172.21.15.19:0/4055503940 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76640443f0 con 0x7f76740a9720 2024-05-04T18:18:58.823 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 1 -- 172.21.15.19:0/4055503940 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76640447e0 con 0x7f76740a9720 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f767aa42640 5 monclient: authenticate success, global_id 15843 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f7670ff9640 10 monclient: dump: 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:58.824 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:58.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f767aa42640 10 monclient: _renew_subs 2024-05-04T18:18:58.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f767aa42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T18:18:58.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f767aa42640 1 -- 172.21.15.19:0/4055503940 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76741ce900 con 0x7f76740a9720 2024-05-04T18:18:58.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f76727fc640 4 set_mon_vals no callback set 2024-05-04T18:18:58.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f767aa42640 10 client.15843.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:58.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f767aa42640 10 monclient: _renew_subs 2024-05-04T18:18:58.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f767aa42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T18:18:58.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f767aa42640 1 -- 172.21.15.19:0/4055503940 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76741cd300 con 0x7f76740a9720 2024-05-04T18:18:58.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.820+0000 7f767aa42640 1 librados: init done 2024-05-04T18:18:58.825 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.821+0000 7f767aa42640 10 client.15843.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:58.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.821+0000 7f767aa42640 15 client.15843.objecter _session_command_op_assign -1 1 2024-05-04T18:18:58.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.821+0000 7f767aa42640 20 client.15843.objecter _get_session osd=-1 returning homeless 2024-05-04T18:18:58.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.821+0000 7f767aa42640 10 client.15843.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:58.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.821+0000 7f767aa42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T18:18:58.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.821+0000 7f767aa42640 1 -- 172.21.15.19:0/4055503940 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f767406a700 con 0x7f76740a9720 2024-05-04T18:18:58.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.841+0000 7f7670ff9640 1 -- 172.21.15.19:0/4055503940 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f766404b060 con 0x7f76740a9720 2024-05-04T18:18:58.842 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.841+0000 7f7670ff9640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f766006d810 0x7f766006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.841+0000 7f76737fe640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f766006d810 0x7f766006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:58.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.841+0000 7f76737fe640 10 monclient: get_auth_request con 0x7f766006d810 auth_method 0 2024-05-04T18:18:58.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.842+0000 7f7670ff9640 10 client.15843.objecter ms_handle_connect 0x7f766006d810 2024-05-04T18:18:58.843 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.842+0000 7f76737fe640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f766006d810 0x7f766006fcd0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7674140a20 tx=0x7f7668009110 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:58.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 1 -- 172.21.15.19:0/4055503940 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f7664079950 con 0x7f76740a9720 2024-05-04T18:18:58.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 10 client.15843.objecter ms_dispatch 0x7f7674000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:58.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 3 client.15843.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:58.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 3 client.15843.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:58.846 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 10 client.15843.objecter checking command 1 2024-05-04T18:18:58.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x7f76600755e0 0x7f7660077a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:58.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 20 client.15843.objecter _get_session s=0x7f7660074f10 osd=7 3 2024-05-04T18:18:58.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 20 client.15843.objecter put_session s=0x7f7660074f10 osd=7 3 2024-05-04T18:18:58.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 15 client.15843.objecter _session_command_op_remove -1 1 2024-05-04T18:18:58.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 20 client.15843.objecter _get_session s=0x7f7660074f10 osd=7 3 2024-05-04T18:18:58.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 20 client.15843.objecter get_session s=0x7f7660074f10 osd=7 3 2024-05-04T18:18:58.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 15 client.15843.objecter _session_command_op_assign 7 1 2024-05-04T18:18:58.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 20 client.15843.objecter put_session s=0x7f7660074f10 osd=7 4 2024-05-04T18:18:58.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 10 client.15843.objecter _send_command 1 2024-05-04T18:18:58.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 1 -- 172.21.15.19:0/4055503940 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7664008f00 con 0x7f76600755e0 2024-05-04T18:18:58.847 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7670ff9640 20 client.15843.objecter dump_active .. 0 homeless 2024-05-04T18:18:58.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.845+0000 7f7678fdf640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x7f76600755e0 0x7f7660077a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:58.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.846+0000 7f7670ff9640 1 -- 172.21.15.19:0/4055503940 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_get_version_reply(handle=1 version=1186) ==== 24+0+0 (secure 0 0 0) 0x7f7664077b50 con 0x7f76740a9720 2024-05-04T18:18:58.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.846+0000 7f7670ff9640 10 monclient: handle_get_version_reply finishing 1 version 1186 2024-05-04T18:18:58.848 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.846+0000 7f7678fdf640 10 monclient: get_auth_request con 0x7f76600755e0 auth_method 0 2024-05-04T18:18:58.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.849+0000 7f7678fdf640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x7f76600755e0 0x7f7660077a80 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:58.850 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.849+0000 7f7670ff9640 10 client.15843.objecter ms_handle_connect 0x7f76600755e0 2024-05-04T18:18:58.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.874+0000 7f7670ff9640 1 -- 172.21.15.19:0/4055503940 <== osd.7 v2:172.21.15.186:6826/1403900459 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7664008f00 con 0x7f76600755e0 2024-05-04T18:18:58.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.874+0000 7f7670ff9640 10 client.15843.objecter ms_dispatch 0x7f7674000b90 command_reply(tid 1: 0 ) 2024-05-04T18:18:58.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.874+0000 7f7670ff9640 10 client.15843.objecter _finish_command 1 = system:0 2024-05-04T18:18:58.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.874+0000 7f7670ff9640 20 client.15843.objecter put_session s=0x7f7660074f10 osd=7 4 2024-05-04T18:18:58.875 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.874+0000 7f7670ff9640 15 client.15843.objecter _session_command_op_remove 7 1 2024-05-04T18:18:58.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.887+0000 7f767aa42640 10 client.15843.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-05-04T18:18:58.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.887+0000 7f767aa42640 15 client.15843.objecter _session_command_op_assign -1 2 2024-05-04T18:18:58.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.887+0000 7f767aa42640 20 client.15843.objecter _get_session s=0x7f7660074f10 osd=7 3 2024-05-04T18:18:58.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.887+0000 7f767aa42640 20 client.15843.objecter put_session s=0x7f7660074f10 osd=7 3 2024-05-04T18:18:58.888 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.887+0000 7f767aa42640 20 client.15843.objecter _get_session s=0x7f7660074f10 osd=7 3 2024-05-04T18:18:58.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.887+0000 7f767aa42640 15 client.15843.objecter _session_command_op_remove -1 2 2024-05-04T18:18:58.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.887+0000 7f767aa42640 20 client.15843.objecter get_session s=0x7f7660074f10 osd=7 3 2024-05-04T18:18:58.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.887+0000 7f767aa42640 15 client.15843.objecter _session_command_op_assign 7 2 2024-05-04T18:18:58.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.887+0000 7f767aa42640 20 client.15843.objecter put_session s=0x7f7660074f10 osd=7 4 2024-05-04T18:18:58.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.887+0000 7f767aa42640 10 client.15843.objecter _send_command 2 2024-05-04T18:18:58.889 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.887+0000 7f767aa42640 1 -- 172.21.15.19:0/4055503940 --> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f76741ccd30 con 0x7f76600755e0 2024-05-04T18:18:58.901 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f7670ff9640 1 -- 172.21.15.19:0/4055503940 <== osd.7 v2:172.21.15.186:6826/1403900459 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f76741ccd30 con 0x7f76600755e0 2024-05-04T18:18:58.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f7670ff9640 10 client.15843.objecter ms_dispatch 0x7f7674000b90 command_reply(tid 2: 0 ) 2024-05-04T18:18:58.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f7670ff9640 10 client.15843.objecter _finish_command 2 = system:0 2024-05-04T18:18:58.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f7670ff9640 20 client.15843.objecter put_session s=0x7f7660074f10 osd=7 4 2024-05-04T18:18:58.902 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f7670ff9640 15 client.15843.objecter _session_command_op_remove 7 2 2024-05-04T18:18:58.902 INFO:teuthology.orchestra.run.smithi019.stdout:{ 2024-05-04T18:18:58.902 INFO:teuthology.orchestra.run.smithi019.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-04T18:18:58.902 INFO:teuthology.orchestra.run.smithi019.stdout:} 2024-05-04T18:18:58.903 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 10 librados: watch_flush enter 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 10 librados: watch_flush exit 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 10 client.15843.objecter close_session for osd.7 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 1 -- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x7f76600755e0 msgr2=0x7f7660077a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:6826/1403900459,v1:172.21.15.186:6827/1403900459] conn(0x7f76600755e0 0x7f7660077a80 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 20 client.15843.objecter put_session s=0x7f7660074f10 osd=7 1 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 20 client.15843.objecter shutdown clearing up homeless session... 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 10 client.15843.objecter successfully canceled tick 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 1 -- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f766006d810 msgr2=0x7f766006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f766006d810 0x7f766006fcd0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7674140a20 tx=0x7f7668009110 comp rx=0 tx=0).stop 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 10 monclient: shutdown 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 1 -- 172.21.15.19:0/4055503940 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76740a9720 msgr2=0x7f76741cc7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 1 --2- 172.21.15.19:0/4055503940 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f76740a9720 0x7f76741cc7f0 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7f7664000c00 tx=0x7f76640046c0 comp rx=0 tx=0).stop 2024-05-04T18:18:58.904 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f7678fdf640 1 -- 172.21.15.19:0/4055503940 reap_dead start 2024-05-04T18:18:58.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 1 -- 172.21.15.19:0/4055503940 shutdown_connections 2024-05-04T18:18:58.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 1 -- 172.21.15.19:0/4055503940 >> 172.21.15.19:0/4055503940 conn(0x7f7674067780 msgr2=0x7f767413d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:58.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 1 -- 172.21.15.19:0/4055503940 shutdown_connections 2024-05-04T18:18:58.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 1 -- 172.21.15.19:0/4055503940 wait complete. 2024-05-04T18:18:58.905 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:58.901+0000 7f767aa42640 1 librados: shutdown 2024-05-04T18:18:58.913 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-04T18:18:59.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.013+0000 7fd592959640 10 monclient: get_monmap_and_config 2024-05-04T18:18:59.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.013+0000 7fd592959640 10 monclient: build_initial_monmap 2024-05-04T18:18:59.015 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.013+0000 7fd592959640 10 monclient: monmap: 2024-05-04T18:18:59.015 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:59.015 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:59.015 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:59.015569+0000 2024-05-04T18:18:59.015 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:59.015569+0000 2024-05-04T18:18:59.015 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:59.015 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:59.015 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:59.015 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:59.016 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:59.016 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:59.016 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:59.016 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:59.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.014+0000 7fd592959640 1 Processor -- start 2024-05-04T18:18:59.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.014+0000 7fd592959640 1 -- start start 2024-05-04T18:18:59.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.014+0000 7fd592959640 10 monclient: init 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd58c071a50 0x7fd58c0af620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 10 monclient(hunting): picked mon.noname-c con 0x7fd58c071a50 addr v2:172.21.15.186:3300/0 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd58c071e30 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c139ed0 0x7fd58c13a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 10 monclient(hunting): picked mon.noname-b con 0x7fd58c139ed0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd58c0a9700 con 0x7fd58c071e30 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd58c140d40 con 0x7fd58c071a50 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fd58c0a99e0 con 0x7fd58c139ed0 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:59.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd592959640 10 monclient(hunting): authenticate will time out at 3361.129891s 2024-05-04T18:18:59.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.015+0000 7fd590ef6640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c139ed0 0x7fd58c13a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:59.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.016+0000 7fd590ef6640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c139ed0 0x7fd58c13a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58302/0 (socket says 172.21.15.19:58302) 2024-05-04T18:18:59.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.016+0000 7fd590ef6640 1 -- 172.21.15.19:0/3891160795 learned_addr learned my addr 172.21.15.19:0/3891160795 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:59.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.016+0000 7fd590ef6640 10 monclient(hunting): get_auth_request con 0x7fd58c139ed0 auth_method 0 2024-05-04T18:18:59.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.016+0000 7fd590ef6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:59.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.016+0000 7fd590ef6640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:59.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.016+0000 7fd590ef6640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:59.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.016+0000 7fd590ef6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:59.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.016+0000 7fd590ef6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:59.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.016+0000 7fd590ef6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:59.019 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.018+0000 7fd590ef6640 10 monclient(hunting): handle_auth_done global_id 18848 payload 274 2024-05-04T18:18:59.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.018+0000 7fd590ef6640 1 -- 172.21.15.19:0/3891160795 >> v2:172.21.15.186:3300/0 conn(0x7fd58c071a50 msgr2=0x7fd58c0af620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:59.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.018+0000 7fd590ef6640 1 --2- 172.21.15.19:0/3891160795 >> v2:172.21.15.186:3300/0 conn(0x7fd58c071a50 0x7fd58c0af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.018+0000 7fd590ef6640 1 -- 172.21.15.19:0/3891160795 >> v1:172.21.15.186:6789/0 conn(0x7fd58c071e30 legacy=0x7fd58c1411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:59.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.018+0000 7fd590ef6640 10 monclient: _finish_hunting 0 2024-05-04T18:18:59.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.018+0000 7fd590ef6640 1 monclient: found mon.noname-b 2024-05-04T18:18:59.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.018+0000 7fd590ef6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:59.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.018+0000 7fd590ef6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T18:18:59.020 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.018+0000 7fd590ef6640 1 -- 172.21.15.19:0/3891160795 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58c1408d0 con 0x7fd58c139ed0 2024-05-04T18:18:59.021 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.020+0000 7fd590ef6640 1 --2- 172.21.15.19:0/3891160795 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c139ed0 0x7fd58c13a360 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0b5770 tx=0x7fd58402beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aeb21c0da3e4486 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:59.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 1 -- 172.21.15.19:0/3891160795 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58402ca50 con 0x7fd58c139ed0 2024-05-04T18:18:59.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:59.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:59.025 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 10 monclient: dump: 2024-05-04T18:18:59.025 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:59.025 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:59.025 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 10 monclient: _finish_auth 0 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 10 monclient: _check_auth_tickets 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 1 -- 172.21.15.19:0/3891160795 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd58402cbf0 con 0x7fd58c139ed0 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 1 -- 172.21.15.19:0/3891160795 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5840343f0 con 0x7fd58c139ed0 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 5 monclient: authenticate success, global_id 18848 2024-05-04T18:18:59.026 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 4 set_mon_vals no callback set 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd58affd640 10 monclient: dump: 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:59.027 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:59.030 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 10 monclient: shutdown 2024-05-04T18:18:59.030 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:59.030 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 1 -- 172.21.15.19:0/3891160795 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c139ed0 msgr2=0x7fd58c13a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:59.030 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 1 --2- 172.21.15.19:0/3891160795 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c139ed0 0x7fd58c13a360 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0b5770 tx=0x7fd58402beb0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.030 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 1 -- 172.21.15.19:0/3891160795 shutdown_connections 2024-05-04T18:18:59.030 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 1 --2- 172.21.15.19:0/3891160795 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c139ed0 0x7fd58c13a360 unknown :-1 s=CLOSED pgs=3173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.030 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 1 --2- 172.21.15.19:0/3891160795 >> v2:172.21.15.186:3300/0 conn(0x7fd58c071a50 0x7fd58c0af620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 1 -- 172.21.15.19:0/3891160795 >> 172.21.15.19:0/3891160795 conn(0x7fd58c0b4b10 msgr2=0x7fd58c0b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 1 -- 172.21.15.19:0/3891160795 shutdown_connections 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.024+0000 7fd592959640 1 -- 172.21.15.19:0/3891160795 wait complete. 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.025+0000 7fd592959640 10 monclient: build_initial_monmap 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.025+0000 7fd592959640 10 monclient: monmap: 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:59.031 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.025+0000 7fd592959640 1 librados: starting msgr at 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.025+0000 7fd592959640 1 librados: starting objecter 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.025+0000 7fd592959640 1 Processor -- start 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 1 -- start start 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 1 librados: setting wanted keys 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 1 librados: calling monclient init 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 10 monclient: init 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 10 monclient: _add_conns ranks=[1,2,0] 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd58c071a50 0x7fd58c134780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 10 monclient(hunting): picked mon.noname-b con 0x7fd58c071a50 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c071e30 0x7fd58c136c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 10 monclient(hunting): picked mon.noname-c con 0x7fd58c071e30 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd58c139ed0 0x7fd58c137180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:59.032 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 10 monclient(hunting): picked mon.noname-a con 0x7fd58c139ed0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c13acb0 con 0x7fd58c139ed0 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c13ab30 con 0x7fd58c071a50 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7fd58c14b460 con 0x7fd58c071e30 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.026+0000 7fd592959640 10 monclient(hunting): authenticate will time out at 3361.141372s 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58b7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c071e30 0x7fd58c136c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd58c071a50 0x7fd58c134780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58b7fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c071e30 0x7fd58c136c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58308/0 (socket says 172.21.15.19:58308) 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58b7fe640 1 -- 172.21.15.19:0/2334704641 learned_addr learned my addr 172.21.15.19:0/2334704641 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58b7fe640 10 monclient(hunting): get_auth_request con 0x7fd58c071e30 auth_method 0 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:59.033 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58b7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:59.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:59.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58bfff640 10 monclient(hunting): get_auth_request con 0x7fd58c071a50 auth_method 0 2024-05-04T18:18:59.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:59.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58bfff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:59.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58bfff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:59.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:59.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:59.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.027+0000 7fd58b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:59.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.028+0000 7fd58bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:59.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.028+0000 7fd58bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:59.034 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.028+0000 7fd58bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:59.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.030+0000 7fd58b7fe640 10 monclient(hunting): handle_auth_done global_id 18851 payload 824 2024-05-04T18:18:59.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.030+0000 7fd58b7fe640 1 -- 172.21.15.19:0/2334704641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd58c071a50 msgr2=0x7fd58c134780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:59.035 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.031+0000 7fd58b7fe640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd58c071a50 0x7fd58c134780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.031+0000 7fd58b7fe640 1 -- 172.21.15.19:0/2334704641 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd58c139ed0 msgr2=0x7fd58c137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:59.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.031+0000 7fd58b7fe640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd58c139ed0 0x7fd58c137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.031+0000 7fd58b7fe640 10 monclient: _finish_hunting 0 2024-05-04T18:18:59.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.031+0000 7fd58b7fe640 1 monclient: found mon.noname-c 2024-05-04T18:18:59.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.031+0000 7fd58b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:59.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.031+0000 7fd58b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.19:3301/0 2024-05-04T18:18:59.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.031+0000 7fd58b7fe640 1 -- 172.21.15.19:0/2334704641 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58c136820 con 0x7fd58c071e30 2024-05-04T18:18:59.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.031+0000 7fd58bfff640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd58c071a50 0x7fd58c134780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-04T18:18:59.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.033+0000 7fd58b7fe640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c071e30 0x7fd58c136c40 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00c440 tx=0x7fd57c00c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:59.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.033+0000 7fd588ff9640 10 client.?.objecter ms_handle_connect 0x7fd58c071e30 2024-05-04T18:18:59.036 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.033+0000 7fd588ff9640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:59.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd588ff9640 1 -- 172.21.15.19:0/2334704641 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c017070 con 0x7fd58c071e30 2024-05-04T18:18:59.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd588ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:59.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd588ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T18:18:59.040 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd588ff9640 10 monclient: dump: 2024-05-04T18:18:59.040 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:59.040 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:59.040 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.040 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.040 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:59.040 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:59.040 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:59.040 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd588ff9640 10 monclient: _finish_auth 0 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd588ff9640 10 monclient: _check_auth_tickets 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd588ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd588ff9640 1 -- 172.21.15.19:0/2334704641 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd57c0035f0 con 0x7fd58c071e30 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd592959640 5 monclient: authenticate success, global_id 18851 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd588ff9640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd592959640 10 monclient: _renew_subs 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd592959640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd592959640 1 -- 172.21.15.19:0/2334704641 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd58c133360 con 0x7fd58c071e30 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd58a7fc640 4 set_mon_vals no callback set 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd592959640 10 client.18851.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd592959640 10 monclient: _renew_subs 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd592959640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:59.041 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd592959640 1 -- 172.21.15.19:0/2334704641 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd58c1376c0 con 0x7fd58c071e30 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.037+0000 7fd592959640 1 librados: init done 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.038+0000 7fd592959640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.038+0000 7fd592959640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.038+0000 7fd592959640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.19:3301/0 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.038+0000 7fd592959640 1 -- 172.21.15.19:0/2334704641 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd58c1379d0 con 0x7fd58c071e30 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.038+0000 7fd588ff9640 1 -- 172.21.15.19:0/2334704641 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c021060 con 0x7fd58c071e30 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.038+0000 7fd588ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.038+0000 7fd588ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.038+0000 7fd588ff9640 10 monclient: dump: 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.042 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:59.043 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:59.043 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:59.043 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:59.043 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:59.043 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:59.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.059+0000 7fd588ff9640 1 -- 172.21.15.19:0/2334704641 <== mon.2 v2:172.21.15.19:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd57c01a070 con 0x7fd58c071e30 2024-05-04T18:18:59.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.060+0000 7fd588ff9640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fd57806d810 0x7fd57806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:59.061 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.060+0000 7fd58bfff640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fd57806d810 0x7fd57806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.060+0000 7fd58bfff640 10 monclient: get_auth_request con 0x7fd57806d810 auth_method 0 2024-05-04T18:18:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.061+0000 7fd588ff9640 10 client.18851.objecter ms_handle_connect 0x7fd57806d810 2024-05-04T18:18:59.062 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.061+0000 7fd58bfff640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fd57806d810 0x7fd57806fcd0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd580009d60 tx=0x7fd580009110 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:59.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.366+0000 7fd588ff9640 1 -- 172.21.15.19:0/2334704641 <== mon.2 v2:172.21.15.19:3301/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7fd57c056ab0 con 0x7fd58c071e30 2024-05-04T18:18:59.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.366+0000 7fd588ff9640 10 client.18851.objecter ms_dispatch 0x7fd58c000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:18:59.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.366+0000 7fd588ff9640 3 client.18851.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:18:59.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.366+0000 7fd588ff9640 3 client.18851.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:18:59.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.367+0000 7fd588ff9640 20 client.18851.objecter dump_active .. 0 homeless 2024-05-04T18:18:59.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.367+0000 7fd588ff9640 1 -- 172.21.15.19:0/2334704641 <== mon.2 v2:172.21.15.19:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd57c0010c0 con 0x7fd58c071e30 2024-05-04T18:18:59.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.367+0000 7fd588ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:59.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.367+0000 7fd588ff9640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:18:59.584 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.583+0000 7fd592959640 1 -- 172.21.15.19:0/2334704641 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fd58c068b10 con 0x7fd57806d810 2024-05-04T18:18:59.585 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.584+0000 7fd588ff9640 1 -- 172.21.15.19:0/2334704641 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd58c068b10 con 0x7fd57806d810 2024-05-04T18:18:59.585 INFO:teuthology.orchestra.run.smithi019.stderr:instructed osd(s) 7 to deep-scrub 2024-05-04T18:18:59.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.585+0000 7fd592959640 10 librados: watch_flush enter 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 10 librados: watch_flush exit 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 20 client.18851.objecter shutdown clearing up homeless session... 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 10 client.18851.objecter successfully canceled tick 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 -- 172.21.15.19:0/2334704641 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fd57806d810 msgr2=0x7fd57806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fd57806d810 0x7fd57806fcd0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd580009d60 tx=0x7fd580009110 comp rx=0 tx=0).stop 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 10 monclient: shutdown 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 -- 172.21.15.19:0/2334704641 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c071e30 msgr2=0x7fd58c136c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c071e30 0x7fd58c136c40 secure :-1 s=READY pgs=3174 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00c440 tx=0x7fd57c00c840 comp rx=0 tx=0).stop 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 -- 172.21.15.19:0/2334704641 shutdown_connections 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7fd57806d810 0x7fd57806fcd0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7fd58c139ed0 0x7fd58c137180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7fd58c071e30 0x7fd58c136c40 unknown :-1 s=CLOSED pgs=3174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 --2- 172.21.15.19:0/2334704641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd58c071a50 0x7fd58c134780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 -- 172.21.15.19:0/2334704641 >> 172.21.15.19:0/2334704641 conn(0x7fd58c0b4b10 msgr2=0x7fd58c1411d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:59.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 -- 172.21.15.19:0/2334704641 shutdown_connections 2024-05-04T18:18:59.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 -- 172.21.15.19:0/2334704641 wait complete. 2024-05-04T18:18:59.589 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.586+0000 7fd592959640 1 librados: shutdown 2024-05-04T18:18:59.600 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T18:18:59.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.698+0000 7f9be3d00640 10 monclient: get_monmap_and_config 2024-05-04T18:18:59.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.698+0000 7f9be3d00640 10 monclient: build_initial_monmap 2024-05-04T18:18:59.700 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.698+0000 7f9be3d00640 10 monclient: monmap: 2024-05-04T18:18:59.700 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:59.700 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:59.700 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:18:59.700450+0000 2024-05-04T18:18:59.700 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:18:59.700450+0000 2024-05-04T18:18:59.700 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:59.700 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:59.700 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:18:59.700 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:18:59.700 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:18:59.701 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:18:59.701 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:18:59.701 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:59.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.699+0000 7f9be3d00640 1 Processor -- start 2024-05-04T18:18:59.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.699+0000 7f9be3d00640 1 -- start start 2024-05-04T18:18:59.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.699+0000 7f9be3d00640 10 monclient: init 2024-05-04T18:18:59.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.699+0000 7f9be3d00640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:59.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.699+0000 7f9be3d00640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-05-04T18:18:59.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.699+0000 7f9be3d00640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f9bdc071b20 0x7f9bdc071f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:59.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be3d00640 10 monclient(hunting): picked mon.noname-a con 0x7f9bdc071b20 addr v2:172.21.15.19:3300/0 2024-05-04T18:18:59.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be3d00640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9bdc072440 addr v1:172.21.15.186:6789/0 2024-05-04T18:18:59.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be3d00640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 0x7f9bdc143540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:59.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be3d00640 10 monclient(hunting): picked mon.noname-b con 0x7f9bdc1430d0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:59.703 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be3d00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bdc071120 con 0x7f9bdc072440 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be3d00640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be3d00640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f9bdc071390 con 0x7f9bdc071b20 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be3d00640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be3d00640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f9bdc06c4c0 con 0x7f9bdc1430d0 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be3d00640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be3d00640 10 monclient(hunting): authenticate will time out at 3361.814692s 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be229d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 0x7f9bdc143540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be1a9c640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f9bdc071b20 0x7f9bdc071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be129b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f9bdc072440 0x7f9bdc13fcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.19:44604/0 (socket says 172.21.15.19:44604) 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be129b640 1 -- 172.21.15.19:0/1226339315 learned_addr learned my addr 172.21.15.19:0/1226339315 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.700+0000 7f9be229d640 10 monclient(hunting): get_auth_request con 0x7f9bdc1430d0 auth_method 0 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be229d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be229d640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be229d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be1a9c640 10 monclient(hunting): get_auth_request con 0x7f9bdc071b20 auth_method 0 2024-05-04T18:18:59.704 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be1a9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:59.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be1a9c640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:59.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be1a9c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:59.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be229d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:59.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be229d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:59.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be229d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:59.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be1a9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:59.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be1a9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:59.705 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.701+0000 7f9be1a9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:59.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be0a9a640 1 -- 172.21.15.19:0/1226339315 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3347505088 0 0) 0x7f9bdc071120 con 0x7f9bdc072440 2024-05-04T18:18:59.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be0a9a640 10 monclient(hunting): _negotiate 2024-05-04T18:18:59.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be0a9a640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:59.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be0a9a640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:59.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be0a9a640 10 monclient(hunting): my global_id is 22810 2024-05-04T18:18:59.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be0a9a640 1 -- 172.21.15.19:0/1226339315 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bc8003520 con 0x7f9bdc072440 2024-05-04T18:18:59.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be229d640 10 monclient(hunting): handle_auth_done global_id 18854 payload 274 2024-05-04T18:18:59.706 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be229d640 1 -- 172.21.15.19:0/1226339315 >> v2:172.21.15.19:3300/0 conn(0x7f9bdc071b20 msgr2=0x7f9bdc071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:18:59.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be229d640 1 --2- 172.21.15.19:0/1226339315 >> v2:172.21.15.19:3300/0 conn(0x7f9bdc071b20 0x7f9bdc071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be229d640 1 -- 172.21.15.19:0/1226339315 >> v1:172.21.15.186:6789/0 conn(0x7f9bdc072440 legacy=0x7f9bdc13fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:59.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be229d640 10 monclient: _finish_hunting 0 2024-05-04T18:18:59.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be229d640 1 monclient: found mon.noname-b 2024-05-04T18:18:59.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be229d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:59.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be229d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.19:3301/0 2024-05-04T18:18:59.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be229d640 1 -- 172.21.15.19:0/1226339315 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc143a80 con 0x7f9bdc1430d0 2024-05-04T18:18:59.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.703+0000 7f9be1a9c640 1 --2- 172.21.15.19:0/1226339315 >> v2:172.21.15.19:3300/0 conn(0x7f9bdc071b20 0x7f9bdc071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-04T18:18:59.707 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.705+0000 7f9be229d640 1 --2- 172.21.15.19:0/1226339315 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 0x7f9bdc143540 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009d90 tx=0x7f9bd402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1af8127b3d2ae784 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:59.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 1 -- 172.21.15.19:0/1226339315 <== mon.2 v2:172.21.15.19:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4039070 con 0x7f9bdc1430d0 2024-05-04T18:18:59.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:59.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:18:59.711 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 10 monclient: dump: 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 10 monclient: _finish_auth 0 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 10 monclient: _check_auth_tickets 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:59.712 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 1 -- 172.21.15.19:0/1226339315 <== mon.2 v2:172.21.15.19:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bd402cc50 con 0x7f9bdc1430d0 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be3d00640 5 monclient: authenticate success, global_id 18854 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be3d00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be3d00640 10 monclient: get_monmap_and_config success 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be3d00640 4 set_mon_vals no callback set 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 1 -- 172.21.15.19:0/1226339315 <== mon.2 v2:172.21.15.19:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4034440 con 0x7f9bdc1430d0 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.709+0000 7f9be0a9a640 10 monclient: dump: 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.713 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:59.714 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:59.714 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:59.714 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:59.714 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:59.714 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:59.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.710+0000 7f9be3d00640 10 monclient: shutdown 2024-05-04T18:18:59.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.710+0000 7f9be3d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:18:59.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.710+0000 7f9be3d00640 1 -- 172.21.15.19:0/1226339315 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 msgr2=0x7f9bdc143540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:59.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.710+0000 7f9be3d00640 1 --2- 172.21.15.19:0/1226339315 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 0x7f9bdc143540 secure :-1 s=READY pgs=3175 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009d90 tx=0x7f9bd402bea0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.710+0000 7f9be3d00640 1 -- 172.21.15.19:0/1226339315 shutdown_connections 2024-05-04T18:18:59.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.710+0000 7f9be3d00640 1 --2- 172.21.15.19:0/1226339315 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 0x7f9bdc143540 unknown :-1 s=CLOSED pgs=3175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.710+0000 7f9be3d00640 1 --2- 172.21.15.19:0/1226339315 >> v2:172.21.15.19:3300/0 conn(0x7f9bdc071b20 0x7f9bdc071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.710+0000 7f9be3d00640 1 -- 172.21.15.19:0/1226339315 >> 172.21.15.19:0/1226339315 conn(0x7f9bdc0679a0 msgr2=0x7f9bdc0b6530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:18:59.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.710+0000 7f9be3d00640 1 -- 172.21.15.19:0/1226339315 shutdown_connections 2024-05-04T18:18:59.718 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.710+0000 7f9be3d00640 1 -- 172.21.15.19:0/1226339315 wait complete. 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.711+0000 7f9be3d00640 10 monclient: build_initial_monmap 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.711+0000 7f9be3d00640 10 monclient: monmap: 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.711+0000 7f9be3d00640 1 librados: starting msgr at 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.711+0000 7f9be3d00640 1 librados: starting objecter 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.711+0000 7f9be3d00640 1 Processor -- start 2024-05-04T18:18:59.719 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.711+0000 7f9be3d00640 1 -- start start 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.711+0000 7f9be3d00640 1 librados: setting wanted keys 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.711+0000 7f9be3d00640 1 librados: calling monclient init 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.711+0000 7f9be3d00640 10 monclient: init 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 10 monclient: _reopen_session rank -1 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 10 monclient: _add_conns ranks=[0,1,2] 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bdc071b20 0x7f9bdc136b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 10 monclient(hunting): picked mon.noname-a con 0x7f9bdc071b20 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bdc072440 0x7f9bdc137410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 10 monclient(hunting): picked mon.noname-b con 0x7f9bdc072440 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 0x7f9bdc0afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 10 monclient(hunting): picked mon.noname-c con 0x7f9bdc1430d0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc14d5e0 con 0x7f9bdc071b20 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:59.720 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc14d460 con 0x7f9bdc072440 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 10 monclient(hunting): start opening mon connection 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f9bdc14d760 con 0x7f9bdc1430d0 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 10 monclient(hunting): _renew_subs 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be3d00640 10 monclient(hunting): authenticate will time out at 3361.826887s 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be229d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 0x7f9bdc0afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be1a9c640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bdc071b20 0x7f9bdc136b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be229d640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 0x7f9bdc0afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:58328/0 (socket says 172.21.15.19:58328) 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be229d640 1 -- 172.21.15.19:0/1449461545 learned_addr learned my addr 172.21.15.19:0/1449461545 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be229d640 10 monclient(hunting): get_auth_request con 0x7f9bdc1430d0 auth_method 0 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be229d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be229d640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be229d640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.712+0000 7f9be129b640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bdc072440 0x7f9bdc137410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be1a9c640 10 monclient(hunting): get_auth_request con 0x7f9bdc071b20 auth_method 0 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be1a9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:59.721 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be1a9c640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be1a9c640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be129b640 10 monclient(hunting): get_auth_request con 0x7f9bdc072440 auth_method 0 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be129b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be129b640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be129b640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be229d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be229d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be229d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be1a9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be1a9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be1a9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be129b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be129b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.713+0000 7f9be129b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:18:59.722 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.716+0000 7f9be1a9c640 10 monclient(hunting): handle_auth_done global_id 15849 payload 824 2024-05-04T18:18:59.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.716+0000 7f9be1a9c640 1 -- 172.21.15.19:0/1449461545 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 msgr2=0x7f9bdc0afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:59.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.716+0000 7f9be1a9c640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 0x7f9bdc0afb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.716+0000 7f9be1a9c640 1 -- 172.21.15.19:0/1449461545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bdc072440 msgr2=0x7f9bdc137410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:18:59.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.716+0000 7f9be1a9c640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bdc072440 0x7f9bdc137410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:18:59.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.716+0000 7f9be1a9c640 10 monclient: _finish_hunting 0 2024-05-04T18:18:59.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.716+0000 7f9be1a9c640 1 monclient: found mon.noname-a 2024-05-04T18:18:59.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.716+0000 7f9be1a9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:18:59.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.716+0000 7f9be1a9c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T18:18:59.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.716+0000 7f9be1a9c640 1 -- 172.21.15.19:0/1449461545 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc0adef0 con 0x7f9bdc071b20 2024-05-04T18:18:59.723 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.716+0000 7f9be229d640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 0x7f9bdc0afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-04T18:18:59.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.718+0000 7f9be1a9c640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bdc071b20 0x7f9bdc136b20 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800e870 tx=0x7f9bd800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:59.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.718+0000 7f9bd27fc640 10 client.?.objecter ms_handle_connect 0x7f9bdc071b20 2024-05-04T18:18:59.724 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.718+0000 7f9bd27fc640 10 client.?.objecter resend_mon_ops 2024-05-04T18:18:59.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 1 -- 172.21.15.19:0/1449461545 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8017070 con 0x7f9bdc071b20 2024-05-04T18:18:59.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:59.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T18:18:59.728 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 10 monclient: dump: 2024-05-04T18:18:59.728 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:59.728 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:59.728 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 10 monclient: _finish_auth 0 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 10 monclient: _check_auth_tickets 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 1 -- 172.21.15.19:0/1449461545 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bd800c940 con 0x7f9bdc071b20 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9be3d00640 5 monclient: authenticate success, global_id 15849 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9be3d00640 10 monclient: _renew_subs 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9be3d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9be3d00640 1 -- 172.21.15.19:0/1449461545 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bdc0ae160 con 0x7f9bdc071b20 2024-05-04T18:18:59.729 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 10 monclient: handle_config config(0 keys) 2024-05-04T18:18:59.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 1 -- 172.21.15.19:0/1449461545 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd800cd10 con 0x7f9bdc071b20 2024-05-04T18:18:59.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:18:59.730 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T18:18:59.731 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd27fc640 10 monclient: dump: 2024-05-04T18:18:59.731 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:18:59.731 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:18:59.731 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.731 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:18:59.731 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:18:59.731 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:18:59.731 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:18:59.731 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:18:59.732 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:18:59.732 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:18:59.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9bd3fff640 4 set_mon_vals no callback set 2024-05-04T18:18:59.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9be3d00640 10 client.15849.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:18:59.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9be3d00640 10 monclient: _renew_subs 2024-05-04T18:18:59.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9be3d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T18:18:59.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9be3d00640 1 -- 172.21.15.19:0/1449461545 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bdc0b00c0 con 0x7f9bdc071b20 2024-05-04T18:18:59.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.726+0000 7f9be3d00640 1 librados: init done 2024-05-04T18:18:59.732 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.727+0000 7f9be3d00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:59.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.728+0000 7f9be3d00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:18:59.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.728+0000 7f9be3d00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.19:3300/0 2024-05-04T18:18:59.733 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.728+0000 7f9be3d00640 1 -- 172.21.15.19:0/1449461545 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bdc068b10 con 0x7f9bdc071b20 2024-05-04T18:18:59.750 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.749+0000 7f9bd27fc640 1 -- 172.21.15.19:0/1449461545 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9bd801a070 con 0x7f9bdc071b20 2024-05-04T18:18:59.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.749+0000 7f9bd27fc640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f9bc006d810 0x7f9bc006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:18:59.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.750+0000 7f9be129b640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f9bc006d810 0x7f9bc006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:18:59.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.750+0000 7f9be129b640 10 monclient: get_auth_request con 0x7f9bc006d810 auth_method 0 2024-05-04T18:18:59.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.750+0000 7f9be129b640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f9bc006d810 0x7f9bc006fcd0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc0b5230 tx=0x7f9bcc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:18:59.751 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:18:59.750+0000 7f9bd27fc640 10 client.15849.objecter ms_handle_connect 0x7f9bc006d810 2024-05-04T18:19:00.017 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.015+0000 7f9bd27fc640 1 -- 172.21.15.19:0/1449461545 <== mon.0 v2:172.21.15.19:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f9bd80577f0 con 0x7f9bdc071b20 2024-05-04T18:19:00.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.016+0000 7f9bd27fc640 10 client.15849.objecter ms_dispatch 0x7f9bdc000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:19:00.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.016+0000 7f9bd27fc640 3 client.15849.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:19:00.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.016+0000 7f9bd27fc640 3 client.15849.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:19:00.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.016+0000 7f9bd27fc640 20 client.15849.objecter dump_active .. 0 homeless 2024-05-04T18:19:00.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.016+0000 7f9bd27fc640 1 -- 172.21.15.19:0/1449461545 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9bd8059750 con 0x7f9bdc071b20 2024-05-04T18:19:00.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.016+0000 7f9bd27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:19:00.018 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.016+0000 7f9bd27fc640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:19:00.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.208+0000 7f9be3d00640 1 -- 172.21.15.19:0/1449461545 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9bdc0b1af0 con 0x7f9bc006d810 2024-05-04T18:19:00.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.211+0000 7f9bd27fc640 1 -- 172.21.15.19:0/1449461545 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+115931 (secure 0 0 0) 0x7f9bdc0b1af0 con 0x7f9bc006d810 2024-05-04T18:19:00.214 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T18:19:00.214 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-05-04T18:19:00.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 10 librados: watch_flush enter 2024-05-04T18:19:00.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 10 librados: watch_flush exit 2024-05-04T18:19:00.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 20 client.15849.objecter shutdown clearing up homeless session... 2024-05-04T18:19:00.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 10 client.15849.objecter successfully canceled tick 2024-05-04T18:19:00.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 -- 172.21.15.19:0/1449461545 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f9bc006d810 msgr2=0x7f9bc006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:00.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f9bc006d810 0x7f9bc006fcd0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc0b5230 tx=0x7f9bcc009040 comp rx=0 tx=0).stop 2024-05-04T18:19:00.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 10 monclient: shutdown 2024-05-04T18:19:00.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:19:00.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 -- 172.21.15.19:0/1449461545 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bdc071b20 msgr2=0x7f9bdc136b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:00.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bdc071b20 0x7f9bdc136b20 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800e870 tx=0x7f9bd800ed30 comp rx=0 tx=0).stop 2024-05-04T18:19:00.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 -- 172.21.15.19:0/1449461545 shutdown_connections 2024-05-04T18:19:00.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f9bc006d810 0x7f9bc006fcd0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:00.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f9bdc1430d0 0x7f9bdc0afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:00.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bdc072440 0x7f9bdc137410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:00.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 --2- 172.21.15.19:0/1449461545 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f9bdc071b20 0x7f9bdc136b20 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:00.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 -- 172.21.15.19:0/1449461545 >> 172.21.15.19:0/1449461545 conn(0x7f9bdc0679a0 msgr2=0x7f9bdc06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:19:00.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 -- 172.21.15.19:0/1449461545 shutdown_connections 2024-05-04T18:19:00.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 -- 172.21.15.19:0/1449461545 wait complete. 2024-05-04T18:19:00.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:00.214+0000 7f9be3d00640 1 librados: shutdown 2024-05-04T18:19:00.227 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":2209,"stamp":"2024-05-04T18:18:59.528716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":537800,"kb_used_data":19448,"kb_used_omap":12,"kb_used_meta":517299,"kb_avail":754436920,"statfs":{"total":773094113280,"available":772543406080,"internally_reserved":0,"allocated":19914752,"data_stored":6709414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":529714736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":76,"apply_latency_ms":76,"commit_latency_ns":76000000,"apply_latency_ns":76000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.095213"},"pg_stats":[{"pgid":"14.1e","version":"0'0","reported_seq":2265,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:49.058619+0000","last_change":"2024-05-04T17:46:46.574386+0000","last_active":"2024-05-04T18:18:49.058619+0000","last_peered":"2024-05-04T18:18:49.058619+0000","last_clean":"2024-05-04T18:18:49.058619+0000","last_became_active":"2024-05-04T17:46:46.518637+0000","last_became_peered":"2024-05-04T17:46:46.518637+0000","last_unstale":"2024-05-04T18:18:49.058619+0000","last_undegraded":"2024-05-04T18:18:49.058619+0000","last_fullsized":"2024-05-04T18:18:49.058619+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":2265,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.467115+0000","last_change":"2024-05-04T18:00:26.429131+0000","last_active":"2024-05-04T18:18:48.467115+0000","last_peered":"2024-05-04T18:18:48.467115+0000","last_clean":"2024-05-04T18:18:48.467115+0000","last_became_active":"2024-05-04T18:00:26.421590+0000","last_became_peered":"2024-05-04T18:00:26.421590+0000","last_unstale":"2024-05-04T18:18:48.467115+0000","last_undegraded":"2024-05-04T18:18:48.467115+0000","last_fullsized":"2024-05-04T18:18:48.467115+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:10:52.211055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":2269,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:49.037307+0000","last_change":"2024-05-04T17:47:40.091354+0000","last_active":"2024-05-04T18:18:49.037307+0000","last_peered":"2024-05-04T18:18:49.037307+0000","last_clean":"2024-05-04T18:18:49.037307+0000","last_became_active":"2024-05-04T17:47:40.078079+0000","last_became_peered":"2024-05-04T17:47:40.078079+0000","last_unstale":"2024-05-04T18:18:49.037307+0000","last_undegraded":"2024-05-04T18:18:49.037307+0000","last_fullsized":"2024-05-04T18:18:49.037307+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:08:09.078519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":2284,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.485428+0000","last_change":"2024-05-04T17:46:53.620558+0000","last_active":"2024-05-04T18:18:48.485428+0000","last_peered":"2024-05-04T18:18:48.485428+0000","last_clean":"2024-05-04T18:18:48.485428+0000","last_became_active":"2024-05-04T17:46:53.602969+0000","last_became_peered":"2024-05-04T17:46:53.602969+0000","last_unstale":"2024-05-04T18:18:48.485428+0000","last_undegraded":"2024-05-04T18:18:48.485428+0000","last_fullsized":"2024-05-04T18:18:48.485428+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:00:17.742135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":2276,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:58.020783+0000","last_change":"2024-05-04T18:18:58.021648+0000","last_active":"2024-05-04T18:18:58.020783+0000","last_peered":"2024-05-04T18:18:58.020783+0000","last_clean":"2024-05-04T18:18:58.020783+0000","last_became_active":"2024-05-04T18:00:25.064688+0000","last_became_peered":"2024-05-04T18:00:25.064688+0000","last_unstale":"2024-05-04T18:18:58.020783+0000","last_undegraded":"2024-05-04T18:18:58.020783+0000","last_fullsized":"2024-05-04T18:18:58.020783+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:58.019071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:58.019071+0000","last_clean_scrub_stamp":"2024-05-04T18:18:58.019071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:53:53.446865+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":2287,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.474703+0000","last_change":"2024-05-04T17:46:53.705592+0000","last_active":"2024-05-04T18:18:48.474703+0000","last_peered":"2024-05-04T18:18:48.474703+0000","last_clean":"2024-05-04T18:18:48.474703+0000","last_became_active":"2024-05-04T17:46:53.682041+0000","last_became_peered":"2024-05-04T17:46:53.682041+0000","last_unstale":"2024-05-04T18:18:48.474703+0000","last_undegraded":"2024-05-04T18:18:48.474703+0000","last_fullsized":"2024-05-04T18:18:48.474703+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:09:58.581895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":2293,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.458427+0000","last_change":"2024-05-04T17:46:35.450534+0000","last_active":"2024-05-04T18:18:48.458427+0000","last_peered":"2024-05-04T18:18:48.458427+0000","last_clean":"2024-05-04T18:18:48.458427+0000","last_became_active":"2024-05-04T17:46:35.438066+0000","last_became_peered":"2024-05-04T17:46:35.438066+0000","last_unstale":"2024-05-04T18:18:48.458427+0000","last_undegraded":"2024-05-04T18:18:48.458427+0000","last_fullsized":"2024-05-04T18:18:48.458427+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:28:24.859168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1","version":"105'7","reported_seq":2271,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.451153+0000","last_change":"2024-05-04T17:46:53.697892+0000","last_active":"2024-05-04T18:18:48.451153+0000","last_peered":"2024-05-04T18:18:48.451153+0000","last_clean":"2024-05-04T18:18:48.451153+0000","last_became_active":"2024-05-04T17:46:53.692194+0000","last_became_peered":"2024-05-04T17:46:53.692194+0000","last_unstale":"2024-05-04T18:18:48.451153+0000","last_undegraded":"2024-05-04T18:18:48.451153+0000","last_fullsized":"2024-05-04T18:18:48.451153+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:29:13.161473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":2284,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.498494+0000","last_change":"2024-05-04T17:46:32.428706+0000","last_active":"2024-05-04T18:18:48.498494+0000","last_peered":"2024-05-04T18:18:48.498494+0000","last_clean":"2024-05-04T18:18:48.498494+0000","last_became_active":"2024-05-04T17:46:32.422070+0000","last_became_peered":"2024-05-04T17:46:32.422070+0000","last_unstale":"2024-05-04T18:18:48.498494+0000","last_undegraded":"2024-05-04T18:18:48.498494+0000","last_fullsized":"2024-05-04T18:18:48.498494+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:08:42.163294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":2244,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:54.307107+0000","last_change":"2024-05-04T17:47:33.488923+0000","last_active":"2024-05-04T18:18:54.307107+0000","last_peered":"2024-05-04T18:18:54.307107+0000","last_clean":"2024-05-04T18:18:54.307107+0000","last_became_active":"2024-05-04T17:47:33.470216+0000","last_became_peered":"2024-05-04T17:47:33.470216+0000","last_unstale":"2024-05-04T18:18:54.307107+0000","last_undegraded":"2024-05-04T18:18:54.307107+0000","last_fullsized":"2024-05-04T18:18:54.307107+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":2258,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:55.993399+0000","last_change":"2024-05-04T18:18:55.994865+0000","last_active":"2024-05-04T18:18:55.993399+0000","last_peered":"2024-05-04T18:18:55.993399+0000","last_clean":"2024-05-04T18:18:55.993399+0000","last_became_active":"2024-05-04T17:46:55.287626+0000","last_became_peered":"2024-05-04T17:46:55.287626+0000","last_unstale":"2024-05-04T18:18:55.993399+0000","last_undegraded":"2024-05-04T18:18:55.993399+0000","last_fullsized":"2024-05-04T18:18:55.993399+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:55.990705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:55.990705+0000","last_clean_scrub_stamp":"2024-05-04T18:18:55.990705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:12:07.135909+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":2284,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.480925+0000","last_change":"2024-05-04T17:46:35.598755+0000","last_active":"2024-05-04T18:18:48.480925+0000","last_peered":"2024-05-04T18:18:48.480925+0000","last_clean":"2024-05-04T18:18:48.480925+0000","last_became_active":"2024-05-04T17:46:35.592584+0000","last_became_peered":"2024-05-04T17:46:35.592584+0000","last_unstale":"2024-05-04T18:18:48.480925+0000","last_undegraded":"2024-05-04T18:18:48.480925+0000","last_fullsized":"2024-05-04T18:18:48.480925+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:35:42.158774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":2251,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:57.342872+0000","last_change":"2024-05-04T18:18:57.343715+0000","last_active":"2024-05-04T18:18:57.342872+0000","last_peered":"2024-05-04T18:18:57.342872+0000","last_clean":"2024-05-04T18:18:57.342872+0000","last_became_active":"2024-05-04T17:47:04.857381+0000","last_became_peered":"2024-05-04T17:47:04.857381+0000","last_unstale":"2024-05-04T18:18:57.342872+0000","last_undegraded":"2024-05-04T18:18:57.342872+0000","last_fullsized":"2024-05-04T18:18:57.342872+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:57.341211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:57.341211+0000","last_clean_scrub_stamp":"2024-05-04T18:18:57.341211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:12:20.737620+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1186'78","reported_seq":2292,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:55.332441+0000","last_change":"2024-05-04T18:18:55.334457+0000","last_active":"2024-05-04T18:18:55.332441+0000","last_peered":"2024-05-04T18:18:55.332441+0000","last_clean":"2024-05-04T18:18:55.332441+0000","last_became_active":"2024-05-04T17:45:17.720848+0000","last_became_peered":"2024-05-04T17:45:17.720848+0000","last_unstale":"2024-05-04T18:18:55.332441+0000","last_undegraded":"2024-05-04T18:18:55.332441+0000","last_fullsized":"2024-05-04T18:18:55.332441+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"1186'78","last_scrub_stamp":"2024-05-04T18:18:55.326505+0000","last_deep_scrub":"1186'78","last_deep_scrub_stamp":"2024-05-04T18:18:55.326505+0000","last_clean_scrub_stamp":"2024-05-04T18:18:55.326505+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:21:18.061749+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2283,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:56.047009+0000","last_change":"2024-05-04T17:45:23.593473+0000","last_active":"2024-05-04T18:18:56.047009+0000","last_peered":"2024-05-04T18:18:56.047009+0000","last_clean":"2024-05-04T18:18:56.047009+0000","last_became_active":"2024-05-04T17:45:23.583946+0000","last_became_peered":"2024-05-04T17:45:23.583946+0000","last_unstale":"2024-05-04T18:18:56.047009+0000","last_undegraded":"2024-05-04T18:18:56.047009+0000","last_fullsized":"2024-05-04T18:18:56.047009+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":2284,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.484160+0000","last_change":"2024-05-04T17:46:45.398622+0000","last_active":"2024-05-04T18:18:48.484160+0000","last_peered":"2024-05-04T18:18:48.484160+0000","last_clean":"2024-05-04T18:18:48.484160+0000","last_became_active":"2024-05-04T17:46:45.387152+0000","last_became_peered":"2024-05-04T17:46:45.387152+0000","last_unstale":"2024-05-04T18:18:48.484160+0000","last_undegraded":"2024-05-04T18:18:48.484160+0000","last_fullsized":"2024-05-04T18:18:48.484160+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:51:33.866454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2293,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:56.656994+0000","last_change":"2024-05-04T18:18:56.657839+0000","last_active":"2024-05-04T18:18:56.656994+0000","last_peered":"2024-05-04T18:18:56.656994+0000","last_clean":"2024-05-04T18:18:56.656994+0000","last_became_active":"2024-05-04T17:45:23.262086+0000","last_became_peered":"2024-05-04T17:45:23.262086+0000","last_unstale":"2024-05-04T18:18:56.656994+0000","last_undegraded":"2024-05-04T18:18:56.656994+0000","last_fullsized":"2024-05-04T18:18:56.656994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:56.655331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:56.655331+0000","last_clean_scrub_stamp":"2024-05-04T18:18:56.655331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:10:29.292115+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":2268,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:49.051563+0000","last_change":"2024-05-04T17:47:14.147039+0000","last_active":"2024-05-04T18:18:49.051563+0000","last_peered":"2024-05-04T18:18:49.051563+0000","last_clean":"2024-05-04T18:18:49.051563+0000","last_became_active":"2024-05-04T17:47:14.130381+0000","last_became_peered":"2024-05-04T17:47:14.130381+0000","last_unstale":"2024-05-04T18:18:49.051563+0000","last_undegraded":"2024-05-04T18:18:49.051563+0000","last_fullsized":"2024-05-04T18:18:49.051563+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2274,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.453524+0000","last_change":"2024-05-04T17:45:23.301096+0000","last_active":"2024-05-04T18:18:48.453524+0000","last_peered":"2024-05-04T18:18:48.453524+0000","last_clean":"2024-05-04T18:18:48.453524+0000","last_became_active":"2024-05-04T17:45:23.265024+0000","last_became_peered":"2024-05-04T17:45:23.265024+0000","last_unstale":"2024-05-04T18:18:48.453524+0000","last_undegraded":"2024-05-04T18:18:48.453524+0000","last_fullsized":"2024-05-04T18:18:48.453524+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:48:10.154363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":2246,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:49.018410+0000","last_change":"2024-05-04T18:00:26.429267+0000","last_active":"2024-05-04T18:18:49.018410+0000","last_peered":"2024-05-04T18:18:49.018410+0000","last_clean":"2024-05-04T18:18:49.018410+0000","last_became_active":"2024-05-04T18:00:26.421083+0000","last_became_peered":"2024-05-04T18:00:26.421083+0000","last_unstale":"2024-05-04T18:18:49.018410+0000","last_undegraded":"2024-05-04T18:18:49.018410+0000","last_fullsized":"2024-05-04T18:18:49.018410+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2284,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:52.709509+0000","last_change":"2024-05-04T17:45:23.491691+0000","last_active":"2024-05-04T18:18:52.709509+0000","last_peered":"2024-05-04T18:18:52.709509+0000","last_clean":"2024-05-04T18:18:52.709509+0000","last_became_active":"2024-05-04T17:45:23.480170+0000","last_became_peered":"2024-05-04T17:45:23.480170+0000","last_unstale":"2024-05-04T18:18:52.709509+0000","last_undegraded":"2024-05-04T18:18:52.709509+0000","last_fullsized":"2024-05-04T18:18:52.709509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":2291,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.461767+0000","last_change":"2024-05-04T17:46:35.456577+0000","last_active":"2024-05-04T18:18:48.461767+0000","last_peered":"2024-05-04T18:18:48.461767+0000","last_clean":"2024-05-04T18:18:48.461767+0000","last_became_active":"2024-05-04T17:46:35.445021+0000","last_became_peered":"2024-05-04T17:46:35.445021+0000","last_unstale":"2024-05-04T18:18:48.461767+0000","last_undegraded":"2024-05-04T18:18:48.461767+0000","last_fullsized":"2024-05-04T18:18:48.461767+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:46:31.130508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2295,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.488630+0000","last_change":"2024-05-04T17:45:23.488141+0000","last_active":"2024-05-04T18:18:48.488630+0000","last_peered":"2024-05-04T18:18:48.488630+0000","last_clean":"2024-05-04T18:18:48.488630+0000","last_became_active":"2024-05-04T17:45:23.481564+0000","last_became_peered":"2024-05-04T17:45:23.481564+0000","last_unstale":"2024-05-04T18:18:48.488630+0000","last_undegraded":"2024-05-04T18:18:48.488630+0000","last_fullsized":"2024-05-04T18:18:48.488630+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:23:41.989802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":2257,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:53.631539+0000","last_change":"2024-05-04T18:18:53.632401+0000","last_active":"2024-05-04T18:18:53.631539+0000","last_peered":"2024-05-04T18:18:53.631539+0000","last_clean":"2024-05-04T18:18:53.631539+0000","last_became_active":"2024-05-04T17:46:37.661672+0000","last_became_peered":"2024-05-04T17:46:37.661672+0000","last_unstale":"2024-05-04T18:18:53.631539+0000","last_undegraded":"2024-05-04T18:18:53.631539+0000","last_fullsized":"2024-05-04T18:18:53.631539+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:53.629818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:53.629818+0000","last_clean_scrub_stamp":"2024-05-04T18:18:53.629818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:48:14.952970+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2292,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:57.607892+0000","last_change":"2024-05-04T18:18:57.609132+0000","last_active":"2024-05-04T18:18:57.607892+0000","last_peered":"2024-05-04T18:18:57.607892+0000","last_clean":"2024-05-04T18:18:57.607892+0000","last_became_active":"2024-05-04T17:45:23.370117+0000","last_became_peered":"2024-05-04T17:45:23.370117+0000","last_unstale":"2024-05-04T18:18:57.607892+0000","last_undegraded":"2024-05-04T18:18:57.607892+0000","last_fullsized":"2024-05-04T18:18:57.607892+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:57.603469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:57.603469+0000","last_clean_scrub_stamp":"2024-05-04T18:18:57.603469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:47:48.986682+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":2270,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:56.990020+0000","last_change":"2024-05-04T18:18:56.990866+0000","last_active":"2024-05-04T18:18:56.990020+0000","last_peered":"2024-05-04T18:18:56.990020+0000","last_clean":"2024-05-04T18:18:56.990020+0000","last_became_active":"2024-05-04T17:47:00.051980+0000","last_became_peered":"2024-05-04T17:47:00.051980+0000","last_unstale":"2024-05-04T18:18:56.990020+0000","last_undegraded":"2024-05-04T18:18:56.990020+0000","last_fullsized":"2024-05-04T18:18:56.990020+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:56.988327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:56.988327+0000","last_clean_scrub_stamp":"2024-05-04T18:18:56.988327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:25:01.501435+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2301,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.494203+0000","last_change":"2024-05-04T17:45:23.713140+0000","last_active":"2024-05-04T18:18:48.494203+0000","last_peered":"2024-05-04T18:18:48.494203+0000","last_clean":"2024-05-04T18:18:48.494203+0000","last_became_active":"2024-05-04T17:45:23.686812+0000","last_became_peered":"2024-05-04T17:45:23.686812+0000","last_unstale":"2024-05-04T18:18:48.494203+0000","last_undegraded":"2024-05-04T18:18:48.494203+0000","last_fullsized":"2024-05-04T18:18:48.494203+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:56:49.147299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":2271,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:49.035349+0000","last_change":"2024-05-04T17:47:40.993471+0000","last_active":"2024-05-04T18:18:49.035349+0000","last_peered":"2024-05-04T18:18:49.035349+0000","last_clean":"2024-05-04T18:18:49.035349+0000","last_became_active":"2024-05-04T17:47:40.988027+0000","last_became_peered":"2024-05-04T17:47:40.988027+0000","last_unstale":"2024-05-04T18:18:49.035349+0000","last_undegraded":"2024-05-04T18:18:49.035349+0000","last_fullsized":"2024-05-04T18:18:49.035349+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:41:52.420496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2279,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:49.062007+0000","last_change":"2024-05-04T17:45:23.672655+0000","last_active":"2024-05-04T18:18:49.062007+0000","last_peered":"2024-05-04T18:18:49.062007+0000","last_clean":"2024-05-04T18:18:49.062007+0000","last_became_active":"2024-05-04T17:45:23.667080+0000","last_became_peered":"2024-05-04T17:45:23.667080+0000","last_unstale":"2024-05-04T18:18:49.062007+0000","last_undegraded":"2024-05-04T18:18:49.062007+0000","last_fullsized":"2024-05-04T18:18:49.062007+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:45:21.636973+0000","last_clean_scrub_stamp":"2024-05-04T17:45:21.636973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":2267,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:49.038924+0000","last_change":"2024-05-04T17:46:40.940632+0000","last_active":"2024-05-04T18:18:49.038924+0000","last_peered":"2024-05-04T18:18:49.038924+0000","last_clean":"2024-05-04T18:18:49.038924+0000","last_became_active":"2024-05-04T17:46:40.915492+0000","last_became_peered":"2024-05-04T17:46:40.915492+0000","last_unstale":"2024-05-04T18:18:49.038924+0000","last_undegraded":"2024-05-04T18:18:49.038924+0000","last_fullsized":"2024-05-04T18:18:49.038924+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":2276,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.467254+0000","last_change":"2024-05-04T17:46:53.791115+0000","last_active":"2024-05-04T18:18:48.467254+0000","last_peered":"2024-05-04T18:18:48.467254+0000","last_clean":"2024-05-04T18:18:48.467254+0000","last_became_active":"2024-05-04T17:46:53.785221+0000","last_became_peered":"2024-05-04T17:46:53.785221+0000","last_unstale":"2024-05-04T18:18:48.467254+0000","last_undegraded":"2024-05-04T18:18:48.467254+0000","last_fullsized":"2024-05-04T18:18:48.467254+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:28:11.181195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":2277,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:49.034378+0000","last_change":"2024-05-04T17:46:43.226742+0000","last_active":"2024-05-04T18:18:49.034378+0000","last_peered":"2024-05-04T18:18:49.034378+0000","last_clean":"2024-05-04T18:18:49.034378+0000","last_became_active":"2024-05-04T17:46:43.193128+0000","last_became_peered":"2024-05-04T17:46:43.193128+0000","last_unstale":"2024-05-04T18:18:49.034378+0000","last_undegraded":"2024-05-04T18:18:49.034378+0000","last_fullsized":"2024-05-04T18:18:49.034378+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:52:43.388505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":2290,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.472841+0000","last_change":"2024-05-04T17:46:42.337619+0000","last_active":"2024-05-04T18:18:48.472841+0000","last_peered":"2024-05-04T18:18:48.472841+0000","last_clean":"2024-05-04T18:18:48.472841+0000","last_became_active":"2024-05-04T17:46:42.332321+0000","last_became_peered":"2024-05-04T17:46:42.332321+0000","last_unstale":"2024-05-04T18:18:48.472841+0000","last_undegraded":"2024-05-04T18:18:48.472841+0000","last_fullsized":"2024-05-04T18:18:48.472841+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:45:24.298610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":2265,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:49.053744+0000","last_change":"2024-05-04T17:47:27.489125+0000","last_active":"2024-05-04T18:18:49.053744+0000","last_peered":"2024-05-04T18:18:49.053744+0000","last_clean":"2024-05-04T18:18:49.053744+0000","last_became_active":"2024-05-04T17:47:27.462872+0000","last_became_peered":"2024-05-04T17:47:27.462872+0000","last_unstale":"2024-05-04T18:18:49.053744+0000","last_undegraded":"2024-05-04T18:18:49.053744+0000","last_fullsized":"2024-05-04T18:18:49.053744+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":2252,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:56.245392+0000","last_change":"2024-05-04T18:18:56.246632+0000","last_active":"2024-05-04T18:18:56.245392+0000","last_peered":"2024-05-04T18:18:56.245392+0000","last_clean":"2024-05-04T18:18:56.245392+0000","last_became_active":"2024-05-04T17:47:04.099185+0000","last_became_peered":"2024-05-04T17:47:04.099185+0000","last_unstale":"2024-05-04T18:18:56.245392+0000","last_undegraded":"2024-05-04T18:18:56.245392+0000","last_fullsized":"2024-05-04T18:18:56.245392+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:56.243660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:56.243660+0000","last_clean_scrub_stamp":"2024-05-04T18:18:56.243660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:10:20.916154+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":2278,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.502941+0000","last_change":"2024-05-04T17:47:15.314211+0000","last_active":"2024-05-04T18:18:48.502941+0000","last_peered":"2024-05-04T18:18:48.502941+0000","last_clean":"2024-05-04T18:18:48.502941+0000","last_became_active":"2024-05-04T17:47:15.307302+0000","last_became_peered":"2024-05-04T17:47:15.307302+0000","last_unstale":"2024-05-04T18:18:48.502941+0000","last_undegraded":"2024-05-04T18:18:48.502941+0000","last_fullsized":"2024-05-04T18:18:48.502941+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:21:22.778712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":2269,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:49.045218+0000","last_change":"2024-05-04T17:47:40.632890+0000","last_active":"2024-05-04T18:18:49.045218+0000","last_peered":"2024-05-04T18:18:49.045218+0000","last_clean":"2024-05-04T18:18:49.045218+0000","last_became_active":"2024-05-04T17:47:40.627379+0000","last_became_peered":"2024-05-04T17:47:40.627379+0000","last_unstale":"2024-05-04T18:18:49.045218+0000","last_undegraded":"2024-05-04T18:18:49.045218+0000","last_fullsized":"2024-05-04T18:18:49.045218+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:24:44.506896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":2252,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:58.278373+0000","last_change":"2024-05-04T18:18:58.279229+0000","last_active":"2024-05-04T18:18:58.278373+0000","last_peered":"2024-05-04T18:18:58.278373+0000","last_clean":"2024-05-04T18:18:58.278373+0000","last_became_active":"2024-05-04T17:47:33.296115+0000","last_became_peered":"2024-05-04T17:47:33.296115+0000","last_unstale":"2024-05-04T18:18:58.278373+0000","last_undegraded":"2024-05-04T18:18:58.278373+0000","last_fullsized":"2024-05-04T18:18:58.278373+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:58.276717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:58.276717+0000","last_clean_scrub_stamp":"2024-05-04T18:18:58.276717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:19:10.998665+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":2278,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:49.022384+0000","last_change":"2024-05-04T17:46:42.745978+0000","last_active":"2024-05-04T18:18:49.022384+0000","last_peered":"2024-05-04T18:18:49.022384+0000","last_clean":"2024-05-04T18:18:49.022384+0000","last_became_active":"2024-05-04T17:46:42.724604+0000","last_became_peered":"2024-05-04T17:46:42.724604+0000","last_unstale":"2024-05-04T18:18:49.022384+0000","last_undegraded":"2024-05-04T18:18:49.022384+0000","last_fullsized":"2024-05-04T18:18:49.022384+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:52:59.865599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":2283,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:48.476601+0000","last_change":"2024-05-04T17:46:45.356387+0000","last_active":"2024-05-04T18:18:48.476601+0000","last_peered":"2024-05-04T18:18:48.476601+0000","last_clean":"2024-05-04T18:18:48.476601+0000","last_became_active":"2024-05-04T17:46:45.347368+0000","last_became_peered":"2024-05-04T17:46:45.347368+0000","last_unstale":"2024-05-04T18:18:48.476601+0000","last_undegraded":"2024-05-04T18:18:48.476601+0000","last_fullsized":"2024-05-04T18:18:48.476601+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:10:48.903095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":2270,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:49.025790+0000","last_change":"2024-05-04T17:47:38.797623+0000","last_active":"2024-05-04T18:18:49.025790+0000","last_peered":"2024-05-04T18:18:49.025790+0000","last_clean":"2024-05-04T18:18:49.025790+0000","last_became_active":"2024-05-04T17:47:38.791956+0000","last_became_peered":"2024-05-04T17:47:38.791956+0000","last_unstale":"2024-05-04T18:18:49.025790+0000","last_undegraded":"2024-05-04T18:18:49.025790+0000","last_fullsized":"2024-05-04T18:18:49.025790+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T17:46:23.918854+0000","last_clean_scrub_stamp":"2024-05-04T17:46:23.918854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T17:47:30.587754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313623,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65976,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305864,"statfs":{"total":96636764160,"available":96569204736,"internally_reserved":0,"allocated":2932736,"data_stored":1219743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":0,"up_from":20,"seq":85899346323,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65824,"kb_used_data":2712,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306016,"statfs":{"total":96636764160,"available":96569360384,"internally_reserved":0,"allocated":2777088,"data_stored":963511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379031,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65832,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306008,"statfs":{"total":96636764160,"available":96569352192,"internally_reserved":0,"allocated":2535424,"data_stored":821908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604379032,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66076,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94305764,"statfs":{"total":96636764160,"available":96569102336,"internally_reserved":0,"allocated":2285568,"data_stored":680002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":16,"seq":68719477135,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69504,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302336,"statfs":{"total":96636764160,"available":96565592064,"internally_reserved":0,"allocated":2285568,"data_stored":680002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542549,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65348,"kb_used_data":2236,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306492,"statfs":{"total":96636764160,"available":96569847808,"internally_reserved":0,"allocated":2289664,"data_stored":680007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542545,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69504,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302336,"statfs":{"total":96636764160,"available":96565592064,"internally_reserved":0,"allocated":2285568,"data_stored":680002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575253,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69736,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302104,"statfs":{"total":96636764160,"available":96565354496,"internally_reserved":0,"allocated":2523136,"data_stored":984239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T18:19:00.232 INFO:tasks.ceph:pgid 14.1e last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.232 INFO:tasks.ceph:pgid 14.1f last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.232 INFO:tasks.ceph:pgid 14.1c last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.232 INFO:tasks.ceph:pgid 14.1d last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.232 INFO:tasks.ceph:pgid 14.3 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.232 INFO:tasks.ceph:pgid 14.0 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.232 INFO:tasks.ceph:pgid 14.1 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.232 INFO:tasks.ceph:pgid 14.6 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.233 INFO:tasks.ceph:pgid 14.7 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.233 INFO:tasks.ceph:pgid 14.5 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.233 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-05-04T17:45:21.636973+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=45, tm_sec=21, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.233 INFO:tasks.ceph:pgid 14.c last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.233 INFO:tasks.ceph:pgid 14.d last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.233 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-05-04T17:45:21.636973+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=45, tm_sec=21, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.233 INFO:tasks.ceph:pgid 14.e last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.233 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-05-04T17:45:21.636973+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=45, tm_sec=21, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.233 INFO:tasks.ceph:pgid 14.8 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.234 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-05-04T17:45:21.636973+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=45, tm_sec=21, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.234 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-05-04T17:45:21.636973+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=45, tm_sec=21, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.234 INFO:tasks.ceph:pgid 14.b last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.234 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-05-04T17:45:21.636973+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=45, tm_sec=21, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.234 INFO:tasks.ceph:pgid 14.19 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.234 INFO:tasks.ceph:pgid 14.18 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.234 INFO:tasks.ceph:pgid 14.1b last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.234 INFO:tasks.ceph:pgid 14.1a last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.234 INFO:tasks.ceph:pgid 14.15 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.235 INFO:tasks.ceph:pgid 14.17 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.235 INFO:tasks.ceph:pgid 14.16 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.235 INFO:tasks.ceph:pgid 14.10 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.235 INFO:tasks.ceph:pgid 14.13 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.235 INFO:tasks.ceph:pgid 14.12 last_scrub_stamp 2024-05-04T17:46:23.918854+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=17, tm_min=46, tm_sec=23, tm_wday=5, tm_yday=125, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=4, tm_hour=18, tm_min=18, tm_sec=51, tm_wday=5, tm_yday=125, tm_isdst=0) 2024-05-04T18:19:00.235 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-04T18:19:00.299 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:55.229 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:00.302 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:55.233 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:00.322 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:34:55.253 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:00.355 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:55.287 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:00.356 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:55.288 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:00.369 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:55.259 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:00.373 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:55.263 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:00.376 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:34:55.307 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:00.391 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:34:55.280 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.214 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:10.144 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.214 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:10.146 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.230 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:10.161 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.257 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:10.146 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.260 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:10.149 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.262 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:10.194 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.266 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:10.198 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.276 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:10.166 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.283 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:10.215 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.833 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:10.723 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.834 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:10.724 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.842 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:10.731 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.933 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:10.864 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.933 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:10.864 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.933 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:10.865 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.934 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:10.865 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.942 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:10.874 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:15.943 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:10.874 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:20.236 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-04T18:19:20.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.340+0000 7f380d6d9640 10 monclient: get_monmap_and_config 2024-05-04T18:19:20.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.340+0000 7f380d6d9640 10 monclient: build_initial_monmap 2024-05-04T18:19:20.340 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.340+0000 7f380d6d9640 10 monclient: monmap: 2024-05-04T18:19:20.340 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:19:20.340 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:19:20.341 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:19:20.340952+0000 2024-05-04T18:19:20.341 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:19:20.340952+0000 2024-05-04T18:19:20.341 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:19:20.341 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:19:20.341 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:19:20.341 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:19:20.341 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:19:20.341 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:19:20.341 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:19:20.341 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:19:20.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.340+0000 7f380d6d9640 1 Processor -- start 2024-05-04T18:19:20.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.340+0000 7f380d6d9640 1 -- start start 2024-05-04T18:19:20.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.340+0000 7f380d6d9640 10 monclient: init 2024-05-04T18:19:20.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 10 monclient: _reopen_session rank -1 2024-05-04T18:19:20.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-05-04T18:19:20.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f38080a9fb0 0x7f38080aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:19:20.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 10 monclient(hunting): picked mon.noname-a con 0x7f38080a9fb0 addr v2:172.21.15.19:3300/0 2024-05-04T18:19:20.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38080b27f0 addr v1:172.21.15.186:6789/0 2024-05-04T18:19:20.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f38081404e0 0x7f38081463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:19:20.344 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 10 monclient(hunting): picked mon.noname-c con 0x7f38081404e0 addr v2:172.21.15.186:3300/0 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38080ab880 con 0x7f38080b27f0 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 10 monclient(hunting): start opening mon connection 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7f3808146930 con 0x7f38080a9fb0 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 10 monclient(hunting): start opening mon connection 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f3808146ab0 con 0x7f38081404e0 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 10 monclient(hunting): _renew_subs 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.341+0000 7f380d6d9640 10 monclient(hunting): authenticate will time out at 3382.455152s 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.342+0000 7f3806ffd640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f38080a9fb0 0x7f38080aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.342+0000 7f38077fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f38081404e0 0x7f38081463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.342+0000 7f3806ffd640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7f38080a9fb0 0x7f38080aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55380/0 (socket says 172.21.15.19:55380) 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.342+0000 7f3806ffd640 1 -- 172.21.15.19:0/1973257654 learned_addr learned my addr 172.21.15.19:0/1973257654 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.342+0000 7f3806ffd640 10 monclient(hunting): get_auth_request con 0x7f38080a9fb0 auth_method 0 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.342+0000 7f3806ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.342+0000 7f3806ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.342+0000 7f3806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:19:20.345 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.342+0000 7f38077fe640 10 monclient(hunting): get_auth_request con 0x7f38081404e0 auth_method 0 2024-05-04T18:19:20.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.342+0000 7f38077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:19:20.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.342+0000 7f38077fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:19:20.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.342+0000 7f38077fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:19:20.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.343+0000 7f3806ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:19:20.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.343+0000 7f3806ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:19:20.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.343+0000 7f3806ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:19:20.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.343+0000 7f38077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:19:20.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.343+0000 7f38077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:19:20.346 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.343+0000 7f38077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:19:20.347 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.344+0000 7f3806ffd640 10 monclient(hunting): handle_auth_done global_id 15852 payload 274 2024-05-04T18:19:20.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.344+0000 7f3806ffd640 1 -- 172.21.15.19:0/1973257654 >> v2:172.21.15.186:3300/0 conn(0x7f38081404e0 msgr2=0x7f38081463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:19:20.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.344+0000 7f3806ffd640 1 --2- 172.21.15.19:0/1973257654 >> v2:172.21.15.186:3300/0 conn(0x7f38081404e0 0x7f38081463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.344+0000 7f3806ffd640 1 -- 172.21.15.19:0/1973257654 >> v1:172.21.15.186:6789/0 conn(0x7f38080b27f0 legacy=0x7f380813fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:19:20.348 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.344+0000 7f3806ffd640 10 monclient: _finish_hunting 0 2024-05-04T18:19:20.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.344+0000 7f3806ffd640 1 monclient: found mon.noname-a 2024-05-04T18:19:20.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.344+0000 7f3806ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:19:20.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.344+0000 7f3806ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.19:3300/0 2024-05-04T18:19:20.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.344+0000 7f3806ffd640 1 -- 172.21.15.19:0/1973257654 --> v2:172.21.15.19:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3808146c60 con 0x7f38080a9fb0 2024-05-04T18:19:20.349 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.346+0000 7f3806ffd640 1 --2- 172.21.15.19:0/1973257654 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38080a9fb0 0x7f38080aa390 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7f38080b2c60 tx=0x7f37f802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd269a732e353339 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:19:20.355 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 1 -- 172.21.15.19:0/1973257654 <== mon.0 v2:172.21.15.19:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f802e070 con 0x7f38080a9fb0 2024-05-04T18:19:20.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:19:20.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-05-04T18:19:20.356 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 10 monclient: dump: 2024-05-04T18:19:20.356 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:19:20.356 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:19:20.356 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.356 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.356 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:19:20.356 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:19:20.356 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:19:20.356 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:19:20.356 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 10 monclient: _finish_auth 0 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 10 monclient: _check_auth_tickets 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 1 -- 172.21.15.19:0/1973257654 <== mon.0 v2:172.21.15.19:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37f8037070 con 0x7f38080a9fb0 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 10 monclient: handle_config config(0 keys) 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 1 -- 172.21.15.19:0/1973257654 <== mon.0 v2:172.21.15.19:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f8031a00 con 0x7f38080a9fb0 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f3805ffb640 10 monclient: dump: 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:19:20.357 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f380d6d9640 5 monclient: authenticate success, global_id 15852 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f380d6d9640 10 monclient: get_monmap_and_config success 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.350+0000 7f380d6d9640 4 set_mon_vals no callback set 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.351+0000 7f380d6d9640 10 monclient: shutdown 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.351+0000 7f380d6d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.351+0000 7f380d6d9640 1 -- 172.21.15.19:0/1973257654 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38080a9fb0 msgr2=0x7f38080aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:20.358 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.351+0000 7f380d6d9640 1 --2- 172.21.15.19:0/1973257654 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38080a9fb0 0x7f38080aa390 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7f38080b2c60 tx=0x7f37f802b880 comp rx=0 tx=0).stop 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.351+0000 7f380d6d9640 1 -- 172.21.15.19:0/1973257654 shutdown_connections 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.351+0000 7f380d6d9640 1 --2- 172.21.15.19:0/1973257654 >> v2:172.21.15.186:3300/0 conn(0x7f38081404e0 0x7f38081463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.351+0000 7f380d6d9640 1 --2- 172.21.15.19:0/1973257654 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38080a9fb0 0x7f38080aa390 unknown :-1 s=CLOSED pgs=3078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.351+0000 7f380d6d9640 1 -- 172.21.15.19:0/1973257654 >> 172.21.15.19:0/1973257654 conn(0x7f38080b4e90 msgr2=0x7f38080b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.351+0000 7f380d6d9640 1 -- 172.21.15.19:0/1973257654 shutdown_connections 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.351+0000 7f380d6d9640 1 -- 172.21.15.19:0/1973257654 wait complete. 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.352+0000 7f380d6d9640 10 monclient: build_initial_monmap 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.352+0000 7f380d6d9640 10 monclient: monmap: 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:19:20.359 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:19:20.360 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:19:20.360 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:19:20.360 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:19:20.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.352+0000 7f380d6d9640 1 librados: starting msgr at 2024-05-04T18:19:20.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.352+0000 7f380d6d9640 1 librados: starting objecter 2024-05-04T18:19:20.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.352+0000 7f380d6d9640 1 Processor -- start 2024-05-04T18:19:20.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.352+0000 7f380d6d9640 1 -- start start 2024-05-04T18:19:20.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.352+0000 7f380d6d9640 1 librados: setting wanted keys 2024-05-04T18:19:20.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.352+0000 7f380d6d9640 1 librados: calling monclient init 2024-05-04T18:19:20.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.352+0000 7f380d6d9640 10 monclient: init 2024-05-04T18:19:20.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 10 monclient: _reopen_session rank -1 2024-05-04T18:19:20.360 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38080a9fb0 0x7f3808142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 10 monclient(hunting): picked mon.noname-b con 0x7f38080a9fb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38080b27f0 0x7f38081413d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 10 monclient(hunting): picked mon.noname-a con 0x7f38080b27f0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f38081404e0 0x7f38081dbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 10 monclient(hunting): picked mon.noname-c con 0x7f38081404e0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 10 monclient(hunting): start opening mon connection 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7f380814d7a0 con 0x7f38080b27f0 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 10 monclient(hunting): start opening mon connection 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f380814d620 con 0x7f38080a9fb0 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 10 monclient(hunting): start opening mon connection 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7f380814d920 con 0x7f38081404e0 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 10 monclient(hunting): _renew_subs 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f380d6d9640 10 monclient(hunting): authenticate will time out at 3382.467108s 2024-05-04T18:19:20.361 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f38067fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38080b27f0 0x7f38081413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.353+0000 7f38077fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f38081404e0 0x7f38081dbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38067fc640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38080b27f0 0x7f38081413d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3300/0 says I am v2:172.21.15.19:55382/0 (socket says 172.21.15.19:55382) 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38077fe640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f38081404e0 0x7f38081dbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.19:3301/0 says I am v2:172.21.15.19:38334/0 (socket says 172.21.15.19:38334) 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38067fc640 1 -- 172.21.15.19:0/1213879624 learned_addr learned my addr 172.21.15.19:0/1213879624 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38067fc640 10 monclient(hunting): get_auth_request con 0x7f38080b27f0 auth_method 0 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38067fc640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38067fc640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38077fe640 10 monclient(hunting): get_auth_request con 0x7f38081404e0 auth_method 0 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38077fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38077fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:19:20.362 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f3806ffd640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38080a9fb0 0x7f3808142150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f3806ffd640 10 monclient(hunting): get_auth_request con 0x7f38080a9fb0 auth_method 0 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f3806ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f3806ffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f3806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.354+0000 7f38077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.355+0000 7f3806ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.355+0000 7f3806ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:19:20.363 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.355+0000 7f3806ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:19:20.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.357+0000 7f3806ffd640 10 monclient(hunting): handle_auth_done global_id 22819 payload 824 2024-05-04T18:19:20.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.357+0000 7f3806ffd640 1 -- 172.21.15.19:0/1213879624 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f38081404e0 msgr2=0x7f38081dbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:20.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.357+0000 7f3806ffd640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f38081404e0 0x7f38081dbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.365 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.357+0000 7f3806ffd640 1 -- 172.21.15.19:0/1213879624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38080b27f0 msgr2=0x7f38081413d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.357+0000 7f3806ffd640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38080b27f0 0x7f38081413d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.357+0000 7f3806ffd640 10 monclient: _finish_hunting 0 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.357+0000 7f3806ffd640 1 monclient: found mon.noname-b 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.357+0000 7f3806ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.357+0000 7f3806ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.357+0000 7f3806ffd640 1 -- 172.21.15.19:0/1213879624 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3808145e40 con 0x7f38080a9fb0 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.359+0000 7f3806ffd640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38080a9fb0 0x7f3808142150 secure :-1 s=READY pgs=5844 cs=0 l=1 rev1=1 crypto rx=0x7f3808073540 tx=0x7f37f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.359+0000 7f37ef7fe640 10 client.?.objecter ms_handle_connect 0x7f38080a9fb0 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.359+0000 7f37ef7fe640 10 client.?.objecter resend_mon_ops 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 1 -- 172.21.15.19:0/1213879624 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f8032670 con 0x7f38080a9fb0 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 10 monclient: dump: 2024-05-04T18:19:20.366 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 10 monclient: _finish_auth 0 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 10 monclient: _check_auth_tickets 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 1 -- 172.21.15.19:0/1213879624 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37f8032850 con 0x7f38080a9fb0 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 10 monclient: handle_config config(0 keys) 2024-05-04T18:19:20.367 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 1 -- 172.21.15.19:0/1213879624 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f8032c40 con 0x7f38080a9fb0 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f37ef7fe640 10 monclient: dump: 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:19:20.368 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f380d6d9640 5 monclient: authenticate success, global_id 22819 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f380d6d9640 10 monclient: _renew_subs 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f380d6d9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f380d6d9640 1 -- 172.21.15.19:0/1213879624 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3808141910 con 0x7f38080a9fb0 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f38057fa640 4 set_mon_vals no callback set 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f380d6d9640 10 client.22819.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f380d6d9640 10 monclient: _renew_subs 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f380d6d9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f380d6d9640 1 -- 172.21.15.19:0/1213879624 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38081dc050 con 0x7f38080a9fb0 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.363+0000 7f380d6d9640 1 librados: init done 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.364+0000 7f380d6d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.364+0000 7f380d6d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.364+0000 7f380d6d9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:19:20.369 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.364+0000 7f380d6d9640 1 -- 172.21.15.19:0/1213879624 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3808068b10 con 0x7f38080a9fb0 2024-05-04T18:19:20.382 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.382+0000 7f37ef7fe640 1 -- 172.21.15.19:0/1213879624 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f37f804c060 con 0x7f38080a9fb0 2024-05-04T18:19:20.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.382+0000 7f37ef7fe640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f37f406d810 0x7f37f406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:19:20.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.382+0000 7f38067fc640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f37f406d810 0x7f37f406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:19:20.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.383+0000 7f38067fc640 10 monclient: get_auth_request con 0x7f37f406d810 auth_method 0 2024-05-04T18:19:20.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.383+0000 7f37ef7fe640 10 client.22819.objecter ms_handle_connect 0x7f37f406d810 2024-05-04T18:19:20.383 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.383+0000 7f38067fc640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f37f406d810 0x7f37f406fcd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f38080b2370 tx=0x7f37fc009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:19:20.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.384+0000 7f37ef7fe640 1 -- 172.21.15.19:0/1213879624 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7f37f8078c30 con 0x7f38080a9fb0 2024-05-04T18:19:20.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.384+0000 7f37ef7fe640 10 client.22819.objecter ms_dispatch 0x7f3808000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:19:20.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.384+0000 7f37ef7fe640 3 client.22819.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:19:20.385 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.384+0000 7f37ef7fe640 3 client.22819.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:19:20.386 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.384+0000 7f37ef7fe640 20 client.22819.objecter dump_active .. 0 homeless 2024-05-04T18:19:20.621 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.621+0000 7f37ef7fe640 1 -- 172.21.15.19:0/1213879624 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f37f8040070 con 0x7f38080a9fb0 2024-05-04T18:19:20.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.621+0000 7f37ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:19:20.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.621+0000 7f37ef7fe640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:19:20.826 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.825+0000 7f380d6d9640 1 -- 172.21.15.19:0/1213879624 --> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f380813aaf0 con 0x7f37f406d810 2024-05-04T18:19:20.829 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.829+0000 7f37ef7fe640 1 -- 172.21.15.19:0/1213879624 <== mgr.4117 v2:172.21.15.186:6800/327322581 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+116200 (secure 0 0 0) 0x7f37fc00f820 con 0x7f37f406d810 2024-05-04T18:19:20.830 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-05-04T18:19:20.831 INFO:teuthology.orchestra.run.smithi019.stderr:dumped all 2024-05-04T18:19:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 10 librados: watch_flush enter 2024-05-04T18:19:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 10 librados: watch_flush exit 2024-05-04T18:19:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 20 client.22819.objecter shutdown clearing up homeless session... 2024-05-04T18:19:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 10 client.22819.objecter successfully canceled tick 2024-05-04T18:19:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 -- 172.21.15.19:0/1213879624 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f37f406d810 msgr2=0x7f37f406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f37f406d810 0x7f37f406fcd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f38080b2370 tx=0x7f37fc009040 comp rx=0 tx=0).stop 2024-05-04T18:19:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 10 monclient: shutdown 2024-05-04T18:19:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:19:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 -- 172.21.15.19:0/1213879624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38080a9fb0 msgr2=0x7f3808142150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38080a9fb0 0x7f3808142150 secure :-1 s=READY pgs=5844 cs=0 l=1 rev1=1 crypto rx=0x7f3808073540 tx=0x7f37f80046c0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 -- 172.21.15.19:0/1213879624 shutdown_connections 2024-05-04T18:19:20.833 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7f37f406d810 0x7f37f406fcd0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7f38081404e0 0x7f38081dbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7f38080b27f0 0x7f38081413d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 --2- 172.21.15.19:0/1213879624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38080a9fb0 0x7f3808142150 unknown :-1 s=CLOSED pgs=5844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 -- 172.21.15.19:0/1213879624 >> 172.21.15.19:0/1213879624 conn(0x7f38080b4e90 msgr2=0x7f38080b7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:19:20.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 -- 172.21.15.19:0/1213879624 shutdown_connections 2024-05-04T18:19:20.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 -- 172.21.15.19:0/1213879624 wait complete. 2024-05-04T18:19:20.834 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.831+0000 7f380d6d9640 1 librados: shutdown 2024-05-04T18:19:20.842 INFO:teuthology.orchestra.run.smithi019.stdout:{"pg_ready":true,"pg_map":{"version":2219,"stamp":"2024-05-04T18:19:19.532919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":540984,"kb_used_data":19448,"kb_used_omap":12,"kb_used_meta":521459,"kb_avail":754433736,"statfs":{"total":773094113280,"available":772540145664,"internally_reserved":0,"allocated":19914752,"data_stored":6709414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":533974576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002452"},"pg_stats":[{"pgid":"14.1e","version":"0'0","reported_seq":2273,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:02.557065+0000","last_change":"2024-05-04T18:19:02.557908+0000","last_active":"2024-05-04T18:19:02.557065+0000","last_peered":"2024-05-04T18:19:02.557065+0000","last_clean":"2024-05-04T18:19:02.557065+0000","last_became_active":"2024-05-04T17:46:46.518637+0000","last_became_peered":"2024-05-04T17:46:46.518637+0000","last_unstale":"2024-05-04T18:19:02.557065+0000","last_undegraded":"2024-05-04T18:19:02.557065+0000","last_fullsized":"2024-05-04T18:19:02.557065+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:02.555419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:02.555419+0000","last_clean_scrub_stamp":"2024-05-04T18:19:02.555419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:53:05.906679+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":2274,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:02.685091+0000","last_change":"2024-05-04T18:19:02.685859+0000","last_active":"2024-05-04T18:19:02.685091+0000","last_peered":"2024-05-04T18:19:02.685091+0000","last_clean":"2024-05-04T18:19:02.685091+0000","last_became_active":"2024-05-04T18:00:26.421590+0000","last_became_peered":"2024-05-04T18:00:26.421590+0000","last_unstale":"2024-05-04T18:19:02.685091+0000","last_undegraded":"2024-05-04T18:19:02.685091+0000","last_fullsized":"2024-05-04T18:19:02.685091+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:02.683500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:02.683500+0000","last_clean_scrub_stamp":"2024-05-04T18:19:02.683500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:54:13.262902+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":2279,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:01.847589+0000","last_change":"2024-05-04T18:19:01.847589+0000","last_active":"2024-05-04T18:19:01.847589+0000","last_peered":"2024-05-04T18:19:01.847589+0000","last_clean":"2024-05-04T18:19:01.847589+0000","last_became_active":"2024-05-04T17:47:40.078079+0000","last_became_peered":"2024-05-04T17:47:40.078079+0000","last_unstale":"2024-05-04T18:19:01.847589+0000","last_undegraded":"2024-05-04T18:19:01.847589+0000","last_fullsized":"2024-05-04T18:19:01.847589+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:01.845180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:01.845180+0000","last_clean_scrub_stamp":"2024-05-04T18:19:01.845180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:30:28.348579+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":2294,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:01.865707+0000","last_change":"2024-05-04T18:19:01.865707+0000","last_active":"2024-05-04T18:19:01.865707+0000","last_peered":"2024-05-04T18:19:01.865707+0000","last_clean":"2024-05-04T18:19:01.865707+0000","last_became_active":"2024-05-04T17:46:53.602969+0000","last_became_peered":"2024-05-04T17:46:53.602969+0000","last_unstale":"2024-05-04T18:19:01.865707+0000","last_undegraded":"2024-05-04T18:19:01.865707+0000","last_fullsized":"2024-05-04T18:19:01.865707+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:01.863036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:01.863036+0000","last_clean_scrub_stamp":"2024-05-04T18:19:01.863036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:53:50.713351+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":2276,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:58.020783+0000","last_change":"2024-05-04T18:18:58.021648+0000","last_active":"2024-05-04T18:18:58.020783+0000","last_peered":"2024-05-04T18:18:58.020783+0000","last_clean":"2024-05-04T18:18:58.020783+0000","last_became_active":"2024-05-04T18:00:25.064688+0000","last_became_peered":"2024-05-04T18:00:25.064688+0000","last_unstale":"2024-05-04T18:18:58.020783+0000","last_undegraded":"2024-05-04T18:18:58.020783+0000","last_fullsized":"2024-05-04T18:18:58.020783+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:58.019071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:58.019071+0000","last_clean_scrub_stamp":"2024-05-04T18:18:58.019071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:53:53.446865+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":2296,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:02.864687+0000","last_change":"2024-05-04T18:19:02.865507+0000","last_active":"2024-05-04T18:19:02.864687+0000","last_peered":"2024-05-04T18:19:02.864687+0000","last_clean":"2024-05-04T18:19:02.864687+0000","last_became_active":"2024-05-04T17:46:53.682041+0000","last_became_peered":"2024-05-04T17:46:53.682041+0000","last_unstale":"2024-05-04T18:19:02.864687+0000","last_undegraded":"2024-05-04T18:19:02.864687+0000","last_fullsized":"2024-05-04T18:19:02.864687+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:02.863037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:02.863037+0000","last_clean_scrub_stamp":"2024-05-04T18:19:02.863037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:48:23.388973+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":2302,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:58.789040+0000","last_change":"2024-05-04T18:18:58.789825+0000","last_active":"2024-05-04T18:18:58.789040+0000","last_peered":"2024-05-04T18:18:58.789040+0000","last_clean":"2024-05-04T18:18:58.789040+0000","last_became_active":"2024-05-04T17:46:35.438066+0000","last_became_peered":"2024-05-04T17:46:35.438066+0000","last_unstale":"2024-05-04T18:18:58.789040+0000","last_undegraded":"2024-05-04T18:18:58.789040+0000","last_fullsized":"2024-05-04T18:18:58.789040+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:58.787384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:58.787384+0000","last_clean_scrub_stamp":"2024-05-04T18:18:58.787384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:34:22.757806+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1","version":"1186'8","reported_seq":2281,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:01.665024+0000","last_change":"2024-05-04T18:19:01.666281+0000","last_active":"2024-05-04T18:19:01.665024+0000","last_peered":"2024-05-04T18:19:01.665024+0000","last_clean":"2024-05-04T18:19:01.665024+0000","last_became_active":"2024-05-04T17:46:53.692194+0000","last_became_peered":"2024-05-04T17:46:53.692194+0000","last_unstale":"2024-05-04T18:19:01.665024+0000","last_undegraded":"2024-05-04T18:19:01.665024+0000","last_fullsized":"2024-05-04T18:19:01.665024+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"1186'8","last_scrub_stamp":"2024-05-04T18:19:01.663405+0000","last_deep_scrub":"1186'8","last_deep_scrub_stamp":"2024-05-04T18:19:01.663405+0000","last_clean_scrub_stamp":"2024-05-04T18:19:01.663405+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:51:46.398661+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":2293,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:59.312512+0000","last_change":"2024-05-04T18:18:59.313276+0000","last_active":"2024-05-04T18:18:59.312512+0000","last_peered":"2024-05-04T18:18:59.312512+0000","last_clean":"2024-05-04T18:18:59.312512+0000","last_became_active":"2024-05-04T17:46:32.422070+0000","last_became_peered":"2024-05-04T17:46:32.422070+0000","last_unstale":"2024-05-04T18:18:59.312512+0000","last_undegraded":"2024-05-04T18:18:59.312512+0000","last_fullsized":"2024-05-04T18:18:59.312512+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:59.310933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:59.310933+0000","last_clean_scrub_stamp":"2024-05-04T18:18:59.310933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:43:17.757504+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":2252,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:59.394608+0000","last_change":"2024-05-04T18:18:59.395468+0000","last_active":"2024-05-04T18:18:59.394608+0000","last_peered":"2024-05-04T18:18:59.394608+0000","last_clean":"2024-05-04T18:18:59.394608+0000","last_became_active":"2024-05-04T17:47:33.470216+0000","last_became_peered":"2024-05-04T17:47:33.470216+0000","last_unstale":"2024-05-04T18:18:59.394608+0000","last_undegraded":"2024-05-04T18:18:59.394608+0000","last_fullsized":"2024-05-04T18:18:59.394608+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:59.392949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:59.392949+0000","last_clean_scrub_stamp":"2024-05-04T18:18:59.392949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:37:51.025433+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":2258,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:55.993399+0000","last_change":"2024-05-04T18:18:55.994865+0000","last_active":"2024-05-04T18:18:55.993399+0000","last_peered":"2024-05-04T18:18:55.993399+0000","last_clean":"2024-05-04T18:18:55.993399+0000","last_became_active":"2024-05-04T17:46:55.287626+0000","last_became_peered":"2024-05-04T17:46:55.287626+0000","last_unstale":"2024-05-04T18:18:55.993399+0000","last_undegraded":"2024-05-04T18:18:55.993399+0000","last_fullsized":"2024-05-04T18:18:55.993399+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:55.990705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:55.990705+0000","last_clean_scrub_stamp":"2024-05-04T18:18:55.990705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:12:07.135909+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":2293,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:00.390099+0000","last_change":"2024-05-04T18:19:00.390926+0000","last_active":"2024-05-04T18:19:00.390099+0000","last_peered":"2024-05-04T18:19:00.390099+0000","last_clean":"2024-05-04T18:19:00.390099+0000","last_became_active":"2024-05-04T17:46:35.592584+0000","last_became_peered":"2024-05-04T17:46:35.592584+0000","last_unstale":"2024-05-04T18:19:00.390099+0000","last_undegraded":"2024-05-04T18:19:00.390099+0000","last_fullsized":"2024-05-04T18:19:00.390099+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:00.388412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:00.388412+0000","last_clean_scrub_stamp":"2024-05-04T18:19:00.388412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:26:08.555948+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":2251,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:57.342872+0000","last_change":"2024-05-04T18:18:57.343715+0000","last_active":"2024-05-04T18:18:57.342872+0000","last_peered":"2024-05-04T18:18:57.342872+0000","last_clean":"2024-05-04T18:18:57.342872+0000","last_became_active":"2024-05-04T17:47:04.857381+0000","last_became_peered":"2024-05-04T17:47:04.857381+0000","last_unstale":"2024-05-04T18:18:57.342872+0000","last_undegraded":"2024-05-04T18:18:57.342872+0000","last_fullsized":"2024-05-04T18:18:57.342872+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:57.341211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:57.341211+0000","last_clean_scrub_stamp":"2024-05-04T18:18:57.341211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:12:20.737620+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1186'78","reported_seq":2292,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:55.332441+0000","last_change":"2024-05-04T18:18:55.334457+0000","last_active":"2024-05-04T18:18:55.332441+0000","last_peered":"2024-05-04T18:18:55.332441+0000","last_clean":"2024-05-04T18:18:55.332441+0000","last_became_active":"2024-05-04T17:45:17.720848+0000","last_became_peered":"2024-05-04T17:45:17.720848+0000","last_unstale":"2024-05-04T18:18:55.332441+0000","last_undegraded":"2024-05-04T18:18:55.332441+0000","last_fullsized":"2024-05-04T18:18:55.332441+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"1186'78","last_scrub_stamp":"2024-05-04T18:18:55.326505+0000","last_deep_scrub":"1186'78","last_deep_scrub_stamp":"2024-05-04T18:18:55.326505+0000","last_clean_scrub_stamp":"2024-05-04T18:18:55.326505+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:21:18.061749+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":2291,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:59.584975+0000","last_change":"2024-05-04T18:18:59.585829+0000","last_active":"2024-05-04T18:18:59.584975+0000","last_peered":"2024-05-04T18:18:59.584975+0000","last_clean":"2024-05-04T18:18:59.584975+0000","last_became_active":"2024-05-04T17:45:23.583946+0000","last_became_peered":"2024-05-04T17:45:23.583946+0000","last_unstale":"2024-05-04T18:18:59.584975+0000","last_undegraded":"2024-05-04T18:18:59.584975+0000","last_fullsized":"2024-05-04T18:18:59.584975+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:59.583290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:59.583290+0000","last_clean_scrub_stamp":"2024-05-04T18:18:59.583290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:29:06.614620+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":2293,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:02.366954+0000","last_change":"2024-05-04T18:19:02.367784+0000","last_active":"2024-05-04T18:19:02.366954+0000","last_peered":"2024-05-04T18:19:02.366954+0000","last_clean":"2024-05-04T18:19:02.366954+0000","last_became_active":"2024-05-04T17:46:45.387152+0000","last_became_peered":"2024-05-04T17:46:45.387152+0000","last_unstale":"2024-05-04T18:19:02.366954+0000","last_undegraded":"2024-05-04T18:19:02.366954+0000","last_fullsized":"2024-05-04T18:19:02.366954+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:02.365255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:02.365255+0000","last_clean_scrub_stamp":"2024-05-04T18:19:02.365255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:03:37.052899+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":2293,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:56.656994+0000","last_change":"2024-05-04T18:18:56.657839+0000","last_active":"2024-05-04T18:18:56.656994+0000","last_peered":"2024-05-04T18:18:56.656994+0000","last_clean":"2024-05-04T18:18:56.656994+0000","last_became_active":"2024-05-04T17:45:23.262086+0000","last_became_peered":"2024-05-04T17:45:23.262086+0000","last_unstale":"2024-05-04T18:18:56.656994+0000","last_undegraded":"2024-05-04T18:18:56.656994+0000","last_fullsized":"2024-05-04T18:18:56.656994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:56.655331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:56.655331+0000","last_clean_scrub_stamp":"2024-05-04T18:18:56.655331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:10:29.292115+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":2276,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:03.619123+0000","last_change":"2024-05-04T18:19:03.619955+0000","last_active":"2024-05-04T18:19:03.619123+0000","last_peered":"2024-05-04T18:19:03.619123+0000","last_clean":"2024-05-04T18:19:03.619123+0000","last_became_active":"2024-05-04T17:47:14.130381+0000","last_became_peered":"2024-05-04T17:47:14.130381+0000","last_unstale":"2024-05-04T18:19:03.619123+0000","last_undegraded":"2024-05-04T18:19:03.619123+0000","last_fullsized":"2024-05-04T18:19:03.619123+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:03.617478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:03.617478+0000","last_clean_scrub_stamp":"2024-05-04T18:19:03.617478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T18:26:11.477137+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2283,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:00.614426+0000","last_change":"2024-05-04T18:19:00.615257+0000","last_active":"2024-05-04T18:19:00.614426+0000","last_peered":"2024-05-04T18:19:00.614426+0000","last_clean":"2024-05-04T18:19:00.614426+0000","last_became_active":"2024-05-04T17:45:23.265024+0000","last_became_peered":"2024-05-04T17:45:23.265024+0000","last_unstale":"2024-05-04T18:19:00.614426+0000","last_undegraded":"2024-05-04T18:19:00.614426+0000","last_fullsized":"2024-05-04T18:19:00.614426+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:00.612748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:00.612748+0000","last_clean_scrub_stamp":"2024-05-04T18:19:00.612748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:37:30.545439+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":2254,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:00.296894+0000","last_change":"2024-05-04T18:19:00.297749+0000","last_active":"2024-05-04T18:19:00.296894+0000","last_peered":"2024-05-04T18:19:00.296894+0000","last_clean":"2024-05-04T18:19:00.296894+0000","last_became_active":"2024-05-04T18:00:26.421083+0000","last_became_peered":"2024-05-04T18:00:26.421083+0000","last_unstale":"2024-05-04T18:19:00.296894+0000","last_undegraded":"2024-05-04T18:19:00.296894+0000","last_fullsized":"2024-05-04T18:19:00.296894+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":697,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:00.295220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:00.295220+0000","last_clean_scrub_stamp":"2024-05-04T18:19:00.295220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:39:36.497971+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2292,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:58.567019+0000","last_change":"2024-05-04T18:18:58.567843+0000","last_active":"2024-05-04T18:18:58.567019+0000","last_peered":"2024-05-04T18:18:58.567019+0000","last_clean":"2024-05-04T18:18:58.567019+0000","last_became_active":"2024-05-04T17:45:23.480170+0000","last_became_peered":"2024-05-04T17:45:23.480170+0000","last_unstale":"2024-05-04T18:18:58.567019+0000","last_undegraded":"2024-05-04T18:18:58.567019+0000","last_fullsized":"2024-05-04T18:18:58.567019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:58.565388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:58.565388+0000","last_clean_scrub_stamp":"2024-05-04T18:18:58.565388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:06:45.868569+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":2301,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:59.829872+0000","last_change":"2024-05-04T18:18:59.829872+0000","last_active":"2024-05-04T18:18:59.829872+0000","last_peered":"2024-05-04T18:18:59.829872+0000","last_clean":"2024-05-04T18:18:59.829872+0000","last_became_active":"2024-05-04T17:46:35.445021+0000","last_became_peered":"2024-05-04T17:46:35.445021+0000","last_unstale":"2024-05-04T18:18:59.829872+0000","last_undegraded":"2024-05-04T18:18:59.829872+0000","last_fullsized":"2024-05-04T18:18:59.829872+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:59.827304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:59.827304+0000","last_clean_scrub_stamp":"2024-05-04T18:18:59.827304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:22:21.770502+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2304,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:58.340982+0000","last_change":"2024-05-04T18:18:58.342635+0000","last_active":"2024-05-04T18:18:58.340982+0000","last_peered":"2024-05-04T18:18:58.340982+0000","last_clean":"2024-05-04T18:18:58.340982+0000","last_became_active":"2024-05-04T17:45:23.481564+0000","last_became_peered":"2024-05-04T17:45:23.481564+0000","last_unstale":"2024-05-04T18:18:58.340982+0000","last_undegraded":"2024-05-04T18:18:58.340982+0000","last_fullsized":"2024-05-04T18:18:58.340982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:58.338417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:58.338417+0000","last_clean_scrub_stamp":"2024-05-04T18:18:58.338417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:32:25.343815+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":2258,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:53.632401+0000","last_change":"2024-05-04T18:18:53.632401+0000","last_active":"2024-05-04T18:18:53.632401+0000","last_peered":"2024-05-04T18:18:53.632401+0000","last_clean":"2024-05-04T18:18:53.632401+0000","last_became_active":"2024-05-04T17:46:37.661672+0000","last_became_peered":"2024-05-04T17:46:37.661672+0000","last_unstale":"2024-05-04T18:18:53.632401+0000","last_undegraded":"2024-05-04T18:18:53.632401+0000","last_fullsized":"2024-05-04T18:18:53.632401+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:53.629818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:53.629818+0000","last_clean_scrub_stamp":"2024-05-04T18:18:53.629818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:48:14.952970+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":2293,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:57.609132+0000","last_change":"2024-05-04T18:18:57.609132+0000","last_active":"2024-05-04T18:18:57.609132+0000","last_peered":"2024-05-04T18:18:57.609132+0000","last_clean":"2024-05-04T18:18:57.609132+0000","last_became_active":"2024-05-04T17:45:23.370117+0000","last_became_peered":"2024-05-04T17:45:23.370117+0000","last_unstale":"2024-05-04T18:18:57.609132+0000","last_undegraded":"2024-05-04T18:18:57.609132+0000","last_fullsized":"2024-05-04T18:18:57.609132+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:57.603469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:57.603469+0000","last_clean_scrub_stamp":"2024-05-04T18:18:57.603469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:47:48.986682+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":2271,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:56.990866+0000","last_change":"2024-05-04T18:18:56.990866+0000","last_active":"2024-05-04T18:18:56.990866+0000","last_peered":"2024-05-04T18:18:56.990866+0000","last_clean":"2024-05-04T18:18:56.990866+0000","last_became_active":"2024-05-04T17:47:00.051980+0000","last_became_peered":"2024-05-04T17:47:00.051980+0000","last_unstale":"2024-05-04T18:18:56.990866+0000","last_undegraded":"2024-05-04T18:18:56.990866+0000","last_fullsized":"2024-05-04T18:18:56.990866+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:56.988327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:56.988327+0000","last_clean_scrub_stamp":"2024-05-04T18:18:56.988327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:25:01.501435+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2310,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:57.827850+0000","last_change":"2024-05-04T18:18:57.828681+0000","last_active":"2024-05-04T18:18:57.827850+0000","last_peered":"2024-05-04T18:18:57.827850+0000","last_clean":"2024-05-04T18:18:57.827850+0000","last_became_active":"2024-05-04T17:45:23.686812+0000","last_became_peered":"2024-05-04T17:45:23.686812+0000","last_unstale":"2024-05-04T18:18:57.827850+0000","last_undegraded":"2024-05-04T18:18:57.827850+0000","last_fullsized":"2024-05-04T18:18:57.827850+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:57.826250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:57.826250+0000","last_clean_scrub_stamp":"2024-05-04T18:18:57.826250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T20:42:18.798371+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":2280,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:03.846048+0000","last_change":"2024-05-04T18:19:03.846826+0000","last_active":"2024-05-04T18:19:03.846048+0000","last_peered":"2024-05-04T18:19:03.846048+0000","last_clean":"2024-05-04T18:19:03.846048+0000","last_became_active":"2024-05-04T17:47:40.988027+0000","last_became_peered":"2024-05-04T17:47:40.988027+0000","last_unstale":"2024-05-04T18:19:03.846048+0000","last_undegraded":"2024-05-04T18:19:03.846048+0000","last_fullsized":"2024-05-04T18:19:03.846048+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:03.844448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:03.844448+0000","last_clean_scrub_stamp":"2024-05-04T18:19:03.844448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T04:58:10.951114+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":2287,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:00.568469+0000","last_change":"2024-05-04T18:19:00.569307+0000","last_active":"2024-05-04T18:19:00.568469+0000","last_peered":"2024-05-04T18:19:00.568469+0000","last_clean":"2024-05-04T18:19:00.568469+0000","last_became_active":"2024-05-04T17:45:23.667080+0000","last_became_peered":"2024-05-04T17:45:23.667080+0000","last_unstale":"2024-05-04T18:19:00.568469+0000","last_undegraded":"2024-05-04T18:19:00.568469+0000","last_fullsized":"2024-05-04T18:19:00.568469+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:00.566811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:00.566811+0000","last_clean_scrub_stamp":"2024-05-04T18:19:00.566811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T21:06:25.005374+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":2275,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:01.555254+0000","last_change":"2024-05-04T18:19:01.556108+0000","last_active":"2024-05-04T18:19:01.555254+0000","last_peered":"2024-05-04T18:19:01.555254+0000","last_clean":"2024-05-04T18:19:01.555254+0000","last_became_active":"2024-05-04T17:46:40.915492+0000","last_became_peered":"2024-05-04T17:46:40.915492+0000","last_unstale":"2024-05-04T18:19:01.555254+0000","last_undegraded":"2024-05-04T18:19:01.555254+0000","last_fullsized":"2024-05-04T18:19:01.555254+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:01.553580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:01.553580+0000","last_clean_scrub_stamp":"2024-05-04T18:19:01.553580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:09:00.458987+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":2285,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:03.368613+0000","last_change":"2024-05-04T18:19:03.369381+0000","last_active":"2024-05-04T18:19:03.368613+0000","last_peered":"2024-05-04T18:19:03.368613+0000","last_clean":"2024-05-04T18:19:03.368613+0000","last_became_active":"2024-05-04T17:46:53.785221+0000","last_became_peered":"2024-05-04T17:46:53.785221+0000","last_unstale":"2024-05-04T18:19:03.368613+0000","last_undegraded":"2024-05-04T18:19:03.368613+0000","last_fullsized":"2024-05-04T18:19:03.368613+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:03.367026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:03.367026+0000","last_clean_scrub_stamp":"2024-05-04T18:19:03.367026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:57:27.078221+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":2286,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:59.825418+0000","last_change":"2024-05-04T18:18:59.826675+0000","last_active":"2024-05-04T18:18:59.825418+0000","last_peered":"2024-05-04T18:18:59.825418+0000","last_clean":"2024-05-04T18:18:59.825418+0000","last_became_active":"2024-05-04T17:46:43.193128+0000","last_became_peered":"2024-05-04T17:46:43.193128+0000","last_unstale":"2024-05-04T18:18:59.825418+0000","last_undegraded":"2024-05-04T18:18:59.825418+0000","last_fullsized":"2024-05-04T18:18:59.825418+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:59.822918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:59.822918+0000","last_clean_scrub_stamp":"2024-05-04T18:18:59.822918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T19:34:41.028403+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":2299,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:00.797300+0000","last_change":"2024-05-04T18:19:00.798137+0000","last_active":"2024-05-04T18:19:00.797300+0000","last_peered":"2024-05-04T18:19:00.797300+0000","last_clean":"2024-05-04T18:19:00.797300+0000","last_became_active":"2024-05-04T17:46:42.332321+0000","last_became_peered":"2024-05-04T17:46:42.332321+0000","last_unstale":"2024-05-04T18:19:00.797300+0000","last_undegraded":"2024-05-04T18:19:00.797300+0000","last_fullsized":"2024-05-04T18:19:00.797300+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:00.795468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:00.795468+0000","last_clean_scrub_stamp":"2024-05-04T18:19:00.795468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T23:58:28.775223+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":2273,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:04.648861+0000","last_change":"2024-05-04T18:19:04.649694+0000","last_active":"2024-05-04T18:19:04.648861+0000","last_peered":"2024-05-04T18:19:04.648861+0000","last_clean":"2024-05-04T18:19:04.648861+0000","last_became_active":"2024-05-04T17:47:27.462872+0000","last_became_peered":"2024-05-04T17:47:27.462872+0000","last_unstale":"2024-05-04T18:19:04.648861+0000","last_undegraded":"2024-05-04T18:19:04.648861+0000","last_fullsized":"2024-05-04T18:19:04.648861+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:04.647174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:04.647174+0000","last_clean_scrub_stamp":"2024-05-04T18:19:04.647174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T02:34:50.177156+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":2253,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:56.246632+0000","last_change":"2024-05-04T18:18:56.246632+0000","last_active":"2024-05-04T18:18:56.246632+0000","last_peered":"2024-05-04T18:18:56.246632+0000","last_clean":"2024-05-04T18:18:56.246632+0000","last_became_active":"2024-05-04T17:47:04.099185+0000","last_became_peered":"2024-05-04T17:47:04.099185+0000","last_unstale":"2024-05-04T18:18:56.246632+0000","last_undegraded":"2024-05-04T18:18:56.246632+0000","last_fullsized":"2024-05-04T18:18:56.246632+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:56.243660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:56.243660+0000","last_clean_scrub_stamp":"2024-05-04T18:18:56.243660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T03:10:20.916154+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":2287,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:03.930143+0000","last_change":"2024-05-04T18:19:03.930897+0000","last_active":"2024-05-04T18:19:03.930143+0000","last_peered":"2024-05-04T18:19:03.930143+0000","last_clean":"2024-05-04T18:19:03.930143+0000","last_became_active":"2024-05-04T17:47:15.307302+0000","last_became_peered":"2024-05-04T17:47:15.307302+0000","last_unstale":"2024-05-04T18:19:03.930143+0000","last_undegraded":"2024-05-04T18:19:03.930143+0000","last_fullsized":"2024-05-04T18:19:03.930143+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:03.928628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:03.928628+0000","last_clean_scrub_stamp":"2024-05-04T18:19:03.928628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:34:44.382005+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":2278,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:02.866314+0000","last_change":"2024-05-04T18:19:02.867263+0000","last_active":"2024-05-04T18:19:02.866314+0000","last_peered":"2024-05-04T18:19:02.866314+0000","last_clean":"2024-05-04T18:19:02.866314+0000","last_became_active":"2024-05-04T17:47:40.627379+0000","last_became_peered":"2024-05-04T17:47:40.627379+0000","last_unstale":"2024-05-04T18:19:02.866314+0000","last_undegraded":"2024-05-04T18:19:02.866314+0000","last_fullsized":"2024-05-04T18:19:02.866314+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:02.864672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:02.864672+0000","last_clean_scrub_stamp":"2024-05-04T18:19:02.864672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T22:41:21.552536+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":2253,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:58.279229+0000","last_change":"2024-05-04T18:18:58.279229+0000","last_active":"2024-05-04T18:18:58.279229+0000","last_peered":"2024-05-04T18:18:58.279229+0000","last_clean":"2024-05-04T18:18:58.279229+0000","last_became_active":"2024-05-04T17:47:33.296115+0000","last_became_peered":"2024-05-04T17:47:33.296115+0000","last_unstale":"2024-05-04T18:18:58.279229+0000","last_undegraded":"2024-05-04T18:18:58.279229+0000","last_fullsized":"2024-05-04T18:18:58.279229+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:58.276717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:58.276717+0000","last_clean_scrub_stamp":"2024-05-04T18:18:58.276717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T00:19:10.998665+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":2287,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:18:58.780417+0000","last_change":"2024-05-04T18:18:58.781249+0000","last_active":"2024-05-04T18:18:58.780417+0000","last_peered":"2024-05-04T18:18:58.780417+0000","last_clean":"2024-05-04T18:18:58.780417+0000","last_became_active":"2024-05-04T17:46:42.724604+0000","last_became_peered":"2024-05-04T17:46:42.724604+0000","last_unstale":"2024-05-04T18:18:58.780417+0000","last_undegraded":"2024-05-04T18:18:58.780417+0000","last_fullsized":"2024-05-04T18:18:58.780417+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:18:58.777236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:18:58.777236+0000","last_clean_scrub_stamp":"2024-05-04T18:18:58.777236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:57:55.354101+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":2293,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:01.394793+0000","last_change":"2024-05-04T18:19:01.394793+0000","last_active":"2024-05-04T18:19:01.394793+0000","last_peered":"2024-05-04T18:19:01.394793+0000","last_clean":"2024-05-04T18:19:01.394793+0000","last_became_active":"2024-05-04T17:46:45.347368+0000","last_became_peered":"2024-05-04T17:46:45.347368+0000","last_unstale":"2024-05-04T18:19:01.394793+0000","last_undegraded":"2024-05-04T18:19:01.394793+0000","last_fullsized":"2024-05-04T18:19:01.394793+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:01.392418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:01.392418+0000","last_clean_scrub_stamp":"2024-05-04T18:19:01.392418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T01:29:00.202768+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":2280,"reported_epoch":1186,"state":"active+clean","last_fresh":"2024-05-04T18:19:00.834452+0000","last_change":"2024-05-04T18:19:00.834452+0000","last_active":"2024-05-04T18:19:00.834452+0000","last_peered":"2024-05-04T18:19:00.834452+0000","last_clean":"2024-05-04T18:19:00.834452+0000","last_became_active":"2024-05-04T17:47:38.791956+0000","last_became_peered":"2024-05-04T17:47:38.791956+0000","last_unstale":"2024-05-04T18:19:00.834452+0000","last_undegraded":"2024-05-04T18:19:00.834452+0000","last_fullsized":"2024-05-04T18:19:00.834452+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-04T18:19:00.831749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-04T18:19:00.831749+0000","last_clean_scrub_stamp":"2024-05-04T18:19:00.831749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-06T05:11:03.178246+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":1,"up_from":21,"seq":90194313627,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70136,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94301704,"statfs":{"total":96636764160,"available":96564944896,"internally_reserved":0,"allocated":2932736,"data_stored":1219743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":20,"seq":85899346327,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65824,"kb_used_data":2712,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306016,"statfs":{"total":96636764160,"available":96569360384,"internally_reserved":0,"allocated":2777088,"data_stored":963511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":19,"seq":81604379035,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65588,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306252,"statfs":{"total":96636764160,"available":96569602048,"internally_reserved":0,"allocated":2535424,"data_stored":821908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":19,"seq":81604379036,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65344,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306496,"statfs":{"total":96636764160,"available":96569851904,"internally_reserved":0,"allocated":2285568,"data_stored":680002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":16,"seq":68719477139,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69504,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302336,"statfs":{"total":96636764160,"available":96565592064,"internally_reserved":0,"allocated":2285568,"data_stored":680002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542553,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65348,"kb_used_data":2236,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306492,"statfs":{"total":96636764160,"available":96569847808,"internally_reserved":0,"allocated":2289664,"data_stored":680007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542549,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69504,"kb_used_data":2232,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302336,"statfs":{"total":96636764160,"available":96565592064,"internally_reserved":0,"allocated":2285568,"data_stored":680002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575257,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69736,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302104,"statfs":{"total":96636764160,"available":96565354496,"internally_reserved":0,"allocated":2523136,"data_stored":984239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-04T18:19:20.846 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-04T18:19:20.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.945+0000 7ff5f5f52640 10 monclient: get_monmap_and_config 2024-05-04T18:19:20.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.945+0000 7ff5f5f52640 10 monclient: build_initial_monmap 2024-05-04T18:19:20.945 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.945+0000 7ff5f5f52640 10 monclient: monmap: 2024-05-04T18:19:20.945 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:19:20.945 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:19:20.946 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T18:19:20.945917+0000 2024-05-04T18:19:20.946 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T18:19:20.945917+0000 2024-05-04T18:19:20.946 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:19:20.946 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:19:20.946 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a-legacy 2024-05-04T18:19:20.946 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-05-04T18:19:20.946 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-b 2024-05-04T18:19:20.946 INFO:teuthology.orchestra.run.smithi019.stderr:3: v2:172.21.15.19:3300/0 mon.noname-a 2024-05-04T18:19:20.946 INFO:teuthology.orchestra.run.smithi019.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-05-04T18:19:20.946 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:19:20.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.945+0000 7ff5f5f52640 1 Processor -- start 2024-05-04T18:19:20.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.945+0000 7ff5f5f52640 1 -- start start 2024-05-04T18:19:20.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.945+0000 7ff5f5f52640 10 monclient: init 2024-05-04T18:19:20.948 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 10 monclient: _reopen_session rank -1 2024-05-04T18:19:20.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-05-04T18:19:20.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff5f00a9fb0 0x7ff5f00aa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:19:20.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 10 monclient(hunting): picked mon.noname-c con 0x7ff5f00a9fb0 addr v2:172.21.15.186:3300/0 2024-05-04T18:19:20.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5f00b27f0 addr v1:172.21.15.19:6789/0 2024-05-04T18:19:20.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7ff5f01404e0 0x7ff5f01463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:19:20.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 10 monclient(hunting): picked mon.noname-a con 0x7ff5f01404e0 addr v2:172.21.15.19:3300/0 2024-05-04T18:19:20.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5f00ab880 con 0x7ff5f00b27f0 2024-05-04T18:19:20.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 10 monclient(hunting): start opening mon connection 2024-05-04T18:19:20.949 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 1 -- --> v2:172.21.15.19:3300/0 -- mon_getmap magic: 0 -- 0x7ff5f0146930 con 0x7ff5f01404e0 2024-05-04T18:19:20.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 10 monclient(hunting): start opening mon connection 2024-05-04T18:19:20.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff5f0146ab0 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 10 monclient(hunting): _renew_subs 2024-05-04T18:19:20.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.946+0000 7ff5f5f52640 10 monclient(hunting): authenticate will time out at 3383.060144s 2024-05-04T18:19:20.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.947+0000 7ff5effff640 1 --2- >> v2:172.21.15.19:3300/0 conn(0x7ff5f01404e0 0x7ff5f01463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:19:20.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.947+0000 7ff5ef7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff5f00a9fb0 0x7ff5f00aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:19:20.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.947+0000 7ff5ef7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff5f00a9fb0 0x7ff5f00aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:47202/0 (socket says 172.21.15.19:47202) 2024-05-04T18:19:20.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.947+0000 7ff5ef7fe640 1 -- 172.21.15.19:0/712822831 learned_addr learned my addr 172.21.15.19:0/712822831 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:19:20.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.947+0000 7ff5ef7fe640 10 monclient(hunting): get_auth_request con 0x7ff5f00a9fb0 auth_method 0 2024-05-04T18:19:20.950 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.947+0000 7ff5ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:19:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.947+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:19:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.947+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:19:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.947+0000 7ff5effff640 10 monclient(hunting): get_auth_request con 0x7ff5f01404e0 auth_method 0 2024-05-04T18:19:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.947+0000 7ff5effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:19:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.947+0000 7ff5effff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:19:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.947+0000 7ff5effff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:19:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.948+0000 7ff5ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:19:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.948+0000 7ff5ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:19:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.948+0000 7ff5ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:19:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.948+0000 7ff5effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:19:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.948+0000 7ff5effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:19:20.951 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.948+0000 7ff5effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:19:20.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.949+0000 7ff5ef7fe640 10 monclient(hunting): handle_auth_done global_id 22822 payload 274 2024-05-04T18:19:20.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.949+0000 7ff5ef7fe640 1 -- 172.21.15.19:0/712822831 >> v2:172.21.15.19:3300/0 conn(0x7ff5f01404e0 msgr2=0x7ff5f01463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-04T18:19:20.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.949+0000 7ff5ef7fe640 1 --2- 172.21.15.19:0/712822831 >> v2:172.21.15.19:3300/0 conn(0x7ff5f01404e0 0x7ff5f01463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.949+0000 7ff5ef7fe640 1 -- 172.21.15.19:0/712822831 >> v1:172.21.15.19:6789/0 conn(0x7ff5f00b27f0 legacy=0x7ff5f013fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:20.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.949+0000 7ff5ef7fe640 10 monclient: _finish_hunting 0 2024-05-04T18:19:20.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.949+0000 7ff5ef7fe640 1 monclient: found mon.noname-c 2024-05-04T18:19:20.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.949+0000 7ff5ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:19:20.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.949+0000 7ff5ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-05-04T18:19:20.952 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.949+0000 7ff5ef7fe640 1 -- 172.21.15.19:0/712822831 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f0146c60 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.953 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.951+0000 7ff5ef7fe640 1 --2- 172.21.15.19:0/712822831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f00a9fb0 0x7ff5f00aa390 secure :-1 s=READY pgs=5845 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00b5f70 tx=0x7ff5e002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cff489e03f64441 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:19:20.955 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 1 -- 172.21.15.19:0/712822831 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0035020 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 10 monclient: dump: 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 10 monclient: _finish_auth 0 2024-05-04T18:19:20.956 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 10 monclient: _check_auth_tickets 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 1 -- 172.21.15.19:0/712822831 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5e0031870 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5f5f52640 5 monclient: authenticate success, global_id 22822 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5f5f52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 10 monclient: handle_config config(0 keys) 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5f5f52640 10 monclient: get_monmap_and_config success 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 1 -- 172.21.15.19:0/712822831 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0031c40 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5f5f52640 4 set_mon_vals no callback set 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.955+0000 7ff5ee7fc640 10 monclient: dump: 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.957 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:19:20.958 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:19:20.958 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:19:20.958 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:19:20.958 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:19:20.958 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:19:20.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.956+0000 7ff5f5f52640 10 monclient: shutdown 2024-05-04T18:19:20.962 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.956+0000 7ff5f5f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.956+0000 7ff5f5f52640 1 -- 172.21.15.19:0/712822831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f00a9fb0 msgr2=0x7ff5f00aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.956+0000 7ff5f5f52640 1 --2- 172.21.15.19:0/712822831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f00a9fb0 0x7ff5f00aa390 secure :-1 s=READY pgs=5845 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00b5f70 tx=0x7ff5e002b910 comp rx=0 tx=0).stop 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.956+0000 7ff5f5f52640 1 -- 172.21.15.19:0/712822831 shutdown_connections 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.956+0000 7ff5f5f52640 1 --2- 172.21.15.19:0/712822831 >> v2:172.21.15.19:3300/0 conn(0x7ff5f01404e0 0x7ff5f01463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.956+0000 7ff5f5f52640 1 --2- 172.21.15.19:0/712822831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f00a9fb0 0x7ff5f00aa390 unknown :-1 s=CLOSED pgs=5845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.956+0000 7ff5f5f52640 1 -- 172.21.15.19:0/712822831 >> 172.21.15.19:0/712822831 conn(0x7ff5f00b4e90 msgr2=0x7ff5f00b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.956+0000 7ff5f5f52640 1 -- 172.21.15.19:0/712822831 shutdown_connections 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.956+0000 7ff5f5f52640 1 -- 172.21.15.19:0/712822831 wait complete. 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.956+0000 7ff5f5f52640 10 monclient: build_initial_monmap 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.956+0000 7ff5f5f52640 10 monclient: monmap: 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-05-04T18:19:20.963 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.noname-a 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.noname-c 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 1 librados: starting msgr at 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 1 librados: starting objecter 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 1 Processor -- start 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 1 -- start start 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 1 librados: setting wanted keys 2024-05-04T18:19:20.964 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 1 librados: calling monclient init 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 10 monclient: init 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 10 monclient: _reopen_session rank -1 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 10 monclient: _add_conns ranks=[1,0,2] 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f00a9fb0 0x7ff5f01db5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 10 monclient(hunting): picked mon.noname-b con 0x7ff5f00a9fb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5f00b27f0 0x7ff5f0140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 10 monclient(hunting): picked mon.noname-a con 0x7ff5f00b27f0 addr [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5f01404e0 0x7ff5f0141c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 10 monclient(hunting): picked mon.noname-c con 0x7ff5f01404e0 addr [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 10 monclient(hunting): start opening mon connection 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 1 -- --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f014d7a0 con 0x7ff5f00b27f0 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 10 monclient(hunting): start opening mon connection 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f014d620 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.965 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 10 monclient(hunting): start opening mon connection 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 1 -- --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_getmap magic: 0 -- 0x7ff5f014d920 con 0x7ff5f01404e0 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 10 monclient(hunting): _renew_subs 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.957+0000 7ff5f5f52640 10 monclient(hunting): authenticate will time out at 3383.071621s 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5ef7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f00a9fb0 0x7ff5f01db5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5effff640 1 --2- >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5f01404e0 0x7ff5f0141c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5eeffd640 1 --2- >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5f00b27f0 0x7ff5f0140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5ef7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f00a9fb0 0x7ff5f01db5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.19:47208/0 (socket says 172.21.15.19:47208) 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5ef7fe640 1 -- 172.21.15.19:0/279442164 learned_addr learned my addr 172.21.15.19:0/279442164 (peer_addr_for_me v2:172.21.15.19:0/0) 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5ef7fe640 10 monclient(hunting): get_auth_request con 0x7ff5f00a9fb0 auth_method 0 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:19:20.966 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5eeffd640 10 monclient(hunting): get_auth_request con 0x7ff5f00b27f0 auth_method 0 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5eeffd640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5effff640 10 monclient(hunting): get_auth_request con 0x7ff5f01404e0 auth_method 0 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5effff640 10 monclient(hunting): _init_auth method 2 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.958+0000 7ff5effff640 10 monclient(hunting): _init_auth creating new auth 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.959+0000 7ff5ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.959+0000 7ff5ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.959+0000 7ff5ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.959+0000 7ff5eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.959+0000 7ff5eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.959+0000 7ff5eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:19:20.967 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.959+0000 7ff5effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-05-04T18:19:20.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.959+0000 7ff5effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-05-04T18:19:20.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.959+0000 7ff5effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-05-04T18:19:20.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.961+0000 7ff5ef7fe640 10 monclient(hunting): handle_auth_done global_id 22825 payload 824 2024-05-04T18:19:20.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.961+0000 7ff5ef7fe640 1 -- 172.21.15.19:0/279442164 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5f01404e0 msgr2=0x7ff5f0141c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:20.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.961+0000 7ff5ef7fe640 1 --2- 172.21.15.19:0/279442164 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5f01404e0 0x7ff5f0141c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.961+0000 7ff5ef7fe640 1 -- 172.21.15.19:0/279442164 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5f00b27f0 msgr2=0x7ff5f0140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:20.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.961+0000 7ff5ef7fe640 1 --2- 172.21.15.19:0/279442164 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5f00b27f0 0x7ff5f0140f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:20.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.961+0000 7ff5ef7fe640 10 monclient: _finish_hunting 0 2024-05-04T18:19:20.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.961+0000 7ff5ef7fe640 1 monclient: found mon.noname-b 2024-05-04T18:19:20.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.961+0000 7ff5ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-05-04T18:19:20.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.961+0000 7ff5ef7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-05-04T18:19:20.968 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.961+0000 7ff5ef7fe640 1 -- 172.21.15.19:0/279442164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f01416f0 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.963+0000 7ff5ef7fe640 1 --2- 172.21.15.19:0/279442164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f00a9fb0 0x7ff5f01db5b0 secure :-1 s=READY pgs=5846 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0000c00 tx=0x7ff5e00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:19:20.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.964+0000 7ff5e7fff640 10 client.?.objecter ms_handle_connect 0x7ff5f00a9fb0 2024-05-04T18:19:20.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.964+0000 7ff5e7fff640 10 client.?.objecter resend_mon_ops 2024-05-04T18:19:20.970 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 1 -- 172.21.15.19:0/279442164 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0040070 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 10 monclient: dump: 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:19:20.971 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 10 monclient: _finish_auth 0 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 10 monclient: _check_auth_tickets 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 1 -- 172.21.15.19:0/279442164 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5e0030d00 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 10 monclient: handle_config config(0 keys) 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 1 -- 172.21.15.19:0/279442164 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0035040 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5e7fff640 10 monclient: dump: 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 18ced2df-a29f-46cc-b71a-1855bfb60c47 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-05-04T17:43:32.318245+0000 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-05-04T18:19:20.972 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:0: [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] mon.a 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2: [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] mon.c 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5f5f52640 5 monclient: authenticate success, global_id 22825 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5f5f52640 10 monclient: _renew_subs 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5f5f52640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5f5f52640 1 -- 172.21.15.19:0/279442164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f013a460 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5edffb640 4 set_mon_vals no callback set 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5f5f52640 10 client.22825.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5f5f52640 10 monclient: _renew_subs 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5f5f52640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5f5f52640 1 -- 172.21.15.19:0/279442164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f01dc140 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.967+0000 7ff5f5f52640 1 librados: init done 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.968+0000 7ff5f5f52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-05-04T18:19:20.973 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.968+0000 7ff5f5f52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:19:20.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.968+0000 7ff5f5f52640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:19:20.974 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.968+0000 7ff5f5f52640 1 -- 172.21.15.19:0/279442164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5f0068b10 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.986+0000 7ff5e7fff640 1 -- 172.21.15.19:0/279442164 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff5e004b060 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.986 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.986+0000 7ff5e7fff640 1 --2- 172.21.15.19:0/279442164 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7ff5d006d810 0x7ff5d006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-04T18:19:20.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.986+0000 7ff5eeffd640 1 --2- 172.21.15.19:0/279442164 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7ff5d006d810 0x7ff5d006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-04T18:19:20.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.986+0000 7ff5eeffd640 10 monclient: get_auth_request con 0x7ff5d006d810 auth_method 0 2024-05-04T18:19:20.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.987+0000 7ff5eeffd640 1 --2- 172.21.15.19:0/279442164 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7ff5d006d810 0x7ff5d006fcd0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00a9a80 tx=0x7ff5d8009040 comp rx=0 tx=0).ready entity=mgr.4117 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-04T18:19:20.987 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.987+0000 7ff5e7fff640 10 client.22825.objecter ms_handle_connect 0x7ff5d006d810 2024-05-04T18:19:20.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.989+0000 7ff5e7fff640 1 -- 172.21.15.19:0/279442164 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(1186..1186 src has 1133..1186) ==== 6412+0+0 (secure 0 0 0) 0x7ff5e0079180 con 0x7ff5f00a9fb0 2024-05-04T18:19:20.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.989+0000 7ff5e7fff640 10 client.22825.objecter ms_dispatch 0x7ff5f0000b90 osd_map(1186..1186 src has 1133..1186) 2024-05-04T18:19:20.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.989+0000 7ff5e7fff640 3 client.22825.objecter handle_osd_map got epochs [1186,1186] > 0 2024-05-04T18:19:20.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.989+0000 7ff5e7fff640 3 client.22825.objecter handle_osd_map decoding full epoch 1186 2024-05-04T18:19:20.989 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:20.989+0000 7ff5e7fff640 20 client.22825.objecter dump_active .. 0 homeless 2024-05-04T18:19:21.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.226+0000 7ff5e7fff640 1 -- 172.21.15.19:0/279442164 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff5e007b0e0 con 0x7ff5f00a9fb0 2024-05-04T18:19:21.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.226+0000 7ff5e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-05-04T18:19:21.226 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.226+0000 7ff5e7fff640 10 monclient: _finish_command 1 = system:0 2024-05-04T18:19:21.415 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.415+0000 7ff5f5f52640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-05-04T18:19:21.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.415+0000 7ff5f5f52640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-05-04T18:19:21.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.415+0000 7ff5f5f52640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-05-04T18:19:21.416 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.415+0000 7ff5f5f52640 1 -- 172.21.15.19:0/279442164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7ff5f0073120 con 0x7ff5f00a9fb0 2024-05-04T18:19:21.608 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.608+0000 7ff5e7fff640 1 -- 172.21.15.19:0/279442164 <== mon.1 v2:172.21.15.186:3300/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ff5e0031870 con 0x7ff5f00a9fb0 2024-05-04T18:19:21.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.608+0000 7ff5e7fff640 10 monclient: handle_config config(1 keys) 2024-05-04T18:19:21.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.608+0000 7ff5edffb640 4 set_mon_vals no callback set 2024-05-04T18:19:21.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.608+0000 7ff5edffb640 10 set_mon_vals mon_health_to_clog = false 2024-05-04T18:19:21.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.608+0000 7ff5e7fff640 1 -- 172.21.15.19:0/279442164 <== mon.1 v2:172.21.15.186:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7ff5e007f3e0 con 0x7ff5f00a9fb0 2024-05-04T18:19:21.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.608+0000 7ff5e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-05-04T18:19:21.609 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.608+0000 7ff5e7fff640 10 monclient: _finish_command 2 = system:0 2024-05-04T18:19:21.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.610+0000 7ff5f5f52640 10 librados: watch_flush enter 2024-05-04T18:19:21.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.610+0000 7ff5f5f52640 10 librados: watch_flush exit 2024-05-04T18:19:21.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.610+0000 7ff5f5f52640 20 client.22825.objecter shutdown clearing up homeless session... 2024-05-04T18:19:21.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.610+0000 7ff5f5f52640 10 client.22825.objecter successfully canceled tick 2024-05-04T18:19:21.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.610+0000 7ff5f5f52640 1 -- 172.21.15.19:0/279442164 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7ff5d006d810 msgr2=0x7ff5d006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:21.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.610+0000 7ff5f5f52640 1 --2- 172.21.15.19:0/279442164 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7ff5d006d810 0x7ff5d006fcd0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00a9a80 tx=0x7ff5d8009040 comp rx=0 tx=0).stop 2024-05-04T18:19:21.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.610+0000 7ff5f5f52640 10 monclient: shutdown 2024-05-04T18:19:21.612 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.610+0000 7ff5f5f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-05-04T18:19:21.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.610+0000 7ff5f5f52640 1 -- 172.21.15.19:0/279442164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f00a9fb0 msgr2=0x7ff5f01db5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-04T18:19:21.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.610+0000 7ff5f5f52640 1 --2- 172.21.15.19:0/279442164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f00a9fb0 0x7ff5f01db5b0 secure :-1 s=READY pgs=5846 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0000c00 tx=0x7ff5e00046c0 comp rx=0 tx=0).stop 2024-05-04T18:19:21.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.611+0000 7ff5f5f52640 1 -- 172.21.15.19:0/279442164 shutdown_connections 2024-05-04T18:19:21.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.611+0000 7ff5f5f52640 1 --2- 172.21.15.19:0/279442164 >> [v2:172.21.15.186:6800/327322581,v1:172.21.15.186:6801/327322581] conn(0x7ff5d006d810 0x7ff5d006fcd0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:21.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.611+0000 7ff5f5f52640 1 --2- 172.21.15.19:0/279442164 >> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] conn(0x7ff5f01404e0 0x7ff5f0141c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:21.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.611+0000 7ff5f5f52640 1 --2- 172.21.15.19:0/279442164 >> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] conn(0x7ff5f00b27f0 0x7ff5f0140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:21.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.611+0000 7ff5f5f52640 1 --2- 172.21.15.19:0/279442164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f00a9fb0 0x7ff5f01db5b0 unknown :-1 s=CLOSED pgs=5846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-04T18:19:21.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.611+0000 7ff5f5f52640 1 -- 172.21.15.19:0/279442164 >> 172.21.15.19:0/279442164 conn(0x7ff5f00b4e90 msgr2=0x7ff5f00aa390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-04T18:19:21.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.611+0000 7ff5f5f52640 1 -- 172.21.15.19:0/279442164 shutdown_connections 2024-05-04T18:19:21.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.611+0000 7ff5f5f52640 1 -- 172.21.15.19:0/279442164 wait complete. 2024-05-04T18:19:21.613 INFO:teuthology.orchestra.run.smithi019.stderr:2024-05-04T18:19:21.611+0000 7ff5f5f52640 1 librados: shutdown 2024-05-04T18:19:21.623 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-04T18:19:21.624 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-04T18:19:21.624 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-04T18:19:21.624 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:19:21.643 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T18:19:21.629+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 33093) UID: 0 2024-05-04T18:19:21.643 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-05-04T18:19:21.631+0000 b485640 -1 osd.0 1186 *** Got signal Terminated *** 2024-05-04T18:19:35.428 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:30.359 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:35.430 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:30.362 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:35.449 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:30.380 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:35.479 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:30.369 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:35.480 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:30.370 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:35.498 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:30.387 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:35.507 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:30.438 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:35.509 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:30.440 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:35.533 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:30.464 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.361 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:50.292 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.366 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:50.297 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.386 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:50.317 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.415 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:50.305 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.416 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:50.347 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.416 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:50.306 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.419 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:50.351 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.434 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:50.323 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.444 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:50.375 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.896 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:50.786 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.897 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:50.787 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.905 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:35:50.795 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.963 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:50.894 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.964 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:50.896 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.968 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:50.900 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.969 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:50.901 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.974 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:35:50.905 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:19:55.979 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:35:50.911 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:15.435 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:10.365 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:15.436 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:10.368 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:15.451 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:10.382 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:15.484 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:10.374 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:15.486 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:10.375 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:15.491 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:10.422 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:15.493 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:10.424 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:15.501 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:10.390 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:15.507 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:10.439 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:16.071 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:10.960 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:16.072 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:10.961 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:16.082 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:10.972 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:16.137 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:11.067 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:16.137 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:11.069 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:16.141 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:11.072 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:16.142 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:11.074 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:16.147 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:11.078 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:16.153 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:11.084 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:35.510 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:30.441 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:35.511 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:30.442 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:35.511 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:30.443 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:35.514 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:30.445 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:35.538 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:30.469 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:35.538 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:30.469 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:35.562 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:30.452 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:35.570 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:30.460 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:35.590 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:30.479 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:36.181 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:31.071 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:36.182 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:31.072 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:36.189 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:31.120 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:36.190 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:31.080 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:36.190 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:31.121 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:36.199 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:31.130 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:36.236 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:31.167 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:36.237 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:31.168 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:36.248 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:31.179 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:55.541 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:50.473 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:55.544 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:50.475 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:55.567 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:50.498 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:55.597 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:50.487 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:55.598 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:50.488 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:55.615 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:50.505 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:55.624 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:50.555 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:55.628 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:50.560 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:55.651 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:50.583 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:56.201 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:51.090 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:56.202 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:51.092 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:56.210 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:51.099 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:56.279 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:51.210 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:56.281 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:51.212 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:56.294 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:51.225 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:56.296 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:51.227 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:56.297 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:51.229 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:56.307 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:51.238 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:20:57.048 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-05-04T18:20:59.247 INFO:tasks.ceph.osd.0:Stopped 2024-05-04T18:20:59.247 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-04T18:20:59.247 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:20:59.261 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T18:20:59.251+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 33100) UID: 0 2024-05-04T18:20:59.261 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-05-04T18:20:59.253+0000 b485640 -1 osd.1 1192 *** Got signal Terminated *** 2024-05-04T18:21:00.510 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:55.441 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:00.564 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:55.495 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:00.576 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:36:55.507 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:00.612 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:55.543 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:00.612 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:55.501 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:00.614 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:55.504 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:00.616 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:55.548 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:00.628 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:36:55.559 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:00.628 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:36:55.518 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:03.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-05-04T18:21:03.157 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-05-04T18:21:05.349 INFO:tasks.ceph.osd.1:Stopped 2024-05-04T18:21:05.349 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-04T18:21:05.350 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:21:05.363 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T18:21:05.354+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 33109) UID: 0 2024-05-04T18:21:05.363 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-05-04T18:21:05.355+0000 b485640 -1 osd.2 1194 *** Got signal Terminated *** 2024-05-04T18:21:09.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-05-04T18:21:10.582 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:05.513 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:10.585 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:05.516 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:10.600 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:05.531 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:10.633 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:05.523 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:10.633 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:05.564 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:10.634 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:05.524 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:10.634 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:05.566 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:10.650 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:05.540 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:10.651 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:05.582 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:15.593 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:10.523 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:15.598 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:10.529 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:15.608 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:10.539 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:15.646 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:10.536 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:15.647 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:10.578 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:15.648 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:10.537 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:15.649 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:10.580 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:15.657 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:10.547 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:15.659 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:10.590 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:28.882 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:23.813 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:28.885 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:23.816 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:28.892 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:23.823 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:28.892 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:23.782 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:28.894 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:23.784 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:28.895 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:23.826 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:28.915 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:23.804 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:28.917 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:23.848 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:28.924 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:23.855 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:30.583 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:25.514 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:30.632 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:25.521 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:30.633 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:25.523 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:30.636 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:25.567 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:30.636 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:25.567 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:30.640 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:25.572 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:30.644 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:25.533 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:30.648 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:25.580 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:30.653 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:25.584 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:35.663 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:30.594 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:35.665 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:30.596 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:35.675 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:30.606 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:35.715 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:30.646 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:35.715 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:30.605 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:35.716 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:30.606 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:35.717 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:30.648 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:35.726 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:30.616 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:35.727 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:30.658 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:50.700 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:45.631 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:50.704 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:45.635 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:50.719 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:45.650 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:50.756 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:45.646 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:50.757 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:45.647 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:50.758 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:45.689 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:50.760 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:45.691 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:50.771 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:45.661 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:50.777 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:45.708 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:55.713 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:50.644 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:55.718 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:50.649 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:55.731 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:37:50.662 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:55.768 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:50.699 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:55.769 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:50.659 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:55.770 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:50.701 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:55.771 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:50.661 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:55.783 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:37:50.715 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:21:55.784 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:37:50.674 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:00.261 INFO:tasks.ceph.osd.2:Stopped 2024-05-04T18:22:00.262 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-04T18:22:00.262 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:22:00.276 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T18:22:00.265+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 33110) UID: 0 2024-05-04T18:22:00.276 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-05-04T18:22:00.266+0000 b485640 -1 osd.3 1197 *** Got signal Terminated *** 2024-05-04T18:22:04.037 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-05-04T18:22:05.772 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:00.703 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:05.775 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:00.706 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:05.797 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:00.728 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:05.826 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:00.716 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:05.827 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:00.717 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:05.831 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:00.762 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:05.834 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:00.765 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:05.843 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:00.733 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:05.848 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:00.779 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:06.364 INFO:tasks.ceph.osd.3:Stopped 2024-05-04T18:22:06.365 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-04T18:22:06.365 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:22:06.377 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T18:22:06.369+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 33075) UID: 0 2024-05-04T18:22:06.378 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-05-04T18:22:06.370+0000 b485640 -1 osd.4 1200 *** Got signal Terminated *** 2024-05-04T18:22:09.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-05-04T18:22:09.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-05-04T18:22:12.466 INFO:tasks.ceph.osd.4:Stopped 2024-05-04T18:22:12.466 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-04T18:22:12.467 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:22:12.483 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-05-04T18:22:12.471+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 33077) UID: 0 2024-05-04T18:22:12.483 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-05-04T18:22:12.472+0000 b485640 -1 osd.5 1202 *** Got signal Terminated *** 2024-05-04T18:22:15.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-05-04T18:22:15.806 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:10.737 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:15.807 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:10.738 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:15.825 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:10.756 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:15.854 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:10.744 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:15.855 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:10.745 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:15.871 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:10.761 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:15.884 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:10.815 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:15.886 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:10.817 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:15.903 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:10.834 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:25.830 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:20.761 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:25.834 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:20.765 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:25.846 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:20.777 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:25.885 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:20.775 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:25.886 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:20.776 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:25.898 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:20.787 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:25.905 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:20.837 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:25.907 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:20.838 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:25.918 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:20.849 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:35.863 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:30.794 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:35.865 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:30.796 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:35.877 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:30.808 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:35.909 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:30.799 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:35.910 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:30.800 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:35.914 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:30.845 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:35.917 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:30.848 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:35.923 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:30.812 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:35.930 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:30.861 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:45.890 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:40.821 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:45.892 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:40.823 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:45.903 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:40.834 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:45.936 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:40.826 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:45.937 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:40.827 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:45.943 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:40.874 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:45.944 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:40.876 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:45.948 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:40.837 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:22:45.955 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:40.886 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:00.927 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:55.858 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:00.931 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:55.863 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:00.946 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:38:55.877 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:00.983 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:55.872 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:00.984 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:55.874 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:00.984 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:55.915 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:00.985 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:55.917 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:00.998 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:38:55.888 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:01.000 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:38:55.932 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:05.945 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:39:00.875 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:05.946 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:39:00.877 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:05.957 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:39:00.888 32798== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:05.992 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:39:00.882 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:05.994 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:39:00.883 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:05.997 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:39:00.928 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:05.998 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:39:00.929 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:06.005 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:39:00.894 32467== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:06.008 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:39:00.940 32806== Warning: unimplemented fcntl command: 1036 2024-05-04T18:23:07.380 INFO:tasks.ceph.osd.5:Stopped 2024-05-04T18:23:07.380 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-04T18:23:07.380 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:23:07.393 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T18:23:07.384+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 33081) UID: 0 2024-05-04T18:23:07.395 INFO:tasks.ceph.osd.6.smithi186.stderr:2024-05-04T18:23:07.385+0000 b485640 -1 osd.6 1204 *** Got signal Terminated *** 2024-05-04T18:23:07.808 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-05-04T18:23:13.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-05-04T18:23:13.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-05-04T18:23:13.481 INFO:tasks.ceph.osd.6:Stopped 2024-05-04T18:23:13.482 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-04T18:23:13.482 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:23:13.501 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T18:23:13.486+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 33083) UID: 0 2024-05-04T18:23:13.501 INFO:tasks.ceph.osd.7.smithi186.stderr:2024-05-04T18:23:13.490+0000 b485640 -1 osd.7 1204 *** Got signal Terminated *** 2024-05-04T18:23:18.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-05-04T18:23:18.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-05-04T18:23:19.584 INFO:tasks.ceph.osd.7:Stopped 2024-05-04T18:23:19.584 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-04T18:23:19.584 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-04T18:23:19.584 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:23:19.610 INFO:tasks.ceph.mgr.y:Stopped 2024-05-04T18:23:19.611 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-04T18:23:19.611 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:23:19.611 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T18:23:19.611+0000 7f33438a2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 32465) UID: 0 2024-05-04T18:23:19.612 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-05-04T18:23:19.611+0000 7f33438a2640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-04T18:23:19.640 INFO:tasks.ceph.mgr.x:Stopped 2024-05-04T18:23:19.640 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-04T18:23:19.640 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-04T18:23:19.640 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:23:19.666 INFO:tasks.ceph.mon.a.smithi019.stderr:2024-05-04T18:23:19.654+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 32774) UID: 0 2024-05-04T18:23:19.666 INFO:tasks.ceph.mon.a.smithi019.stderr:2024-05-04T18:23:19.656+0000 9f9e640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-04T18:23:20.340 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:39:15.271 32798== 2024-05-04T18:23:20.341 INFO:tasks.ceph.mon.a.smithi019.stderr:==00:00:39:15.271 32798== Exit program on first error (--exit-on-first-error=yes) 2024-05-04T18:23:20.365 INFO:tasks.ceph.mon.a.smithi019.stderr:daemon-helper: command failed with exit status 42 2024-05-04T18:23:24.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-05-04T18:23:24.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-05-04T18:23:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-05-04T18:23:25.742 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-05-04T18:23:25.743 INFO:tasks.ceph.mon.a:Stopped 2024-05-04T18:23:25.743 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-04T18:23:25.743 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:23:25.765 INFO:tasks.ceph.mon.c.smithi019.stderr:2024-05-04T18:23:25.747+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 32776) UID: 0 2024-05-04T18:23:25.765 INFO:tasks.ceph.mon.c.smithi019.stderr:2024-05-04T18:23:25.748+0000 9f9e640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-05-04T18:23:26.564 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:39:21.495 32806== 2024-05-04T18:23:26.564 INFO:tasks.ceph.mon.c.smithi019.stderr:==00:00:39:21.495 32806== Exit program on first error (--exit-on-first-error=yes) 2024-05-04T18:23:26.584 INFO:tasks.ceph.mon.c.smithi019.stderr:daemon-helper: command failed with exit status 42 2024-05-04T18:23:29.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-05-04T18:23:29.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-05-04T18:23:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-05-04T18:23:31.845 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-05-04T18:23:31.845 INFO:tasks.ceph.mon.c:Stopped 2024-05-04T18:23:31.845 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-04T18:23:31.845 INFO:teuthology.orchestra.run:waiting for 300 2024-05-04T18:23:31.861 INFO:tasks.ceph.mon.b.smithi186.stderr:2024-05-04T18:23:31.850+0000 9f9e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 32462) UID: 0 2024-05-04T18:23:31.862 INFO:tasks.ceph.mon.b.smithi186.stderr:2024-05-04T18:23:31.851+0000 9f9e640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-05-04T18:23:32.626 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:39:27.516 32467== 2024-05-04T18:23:32.626 INFO:tasks.ceph.mon.b.smithi186.stderr:==00:00:39:27.516 32467== Exit program on first error (--exit-on-first-error=yes) 2024-05-04T18:23:32.645 INFO:tasks.ceph.mon.b.smithi186.stderr:daemon-helper: command failed with exit status 42 2024-05-04T18:23:34.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-05-04T18:23:34.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-05-04T18:23:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-05-04T18:23:37.947 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi186 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-05-04T18:23:37.948 INFO:tasks.ceph.mon.b:Stopped 2024-05-04T18:23:37.948 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-04T18:23:37.948 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-05-04T18:23:37.986 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi019.front.sepia.ceph.com 2024-05-04T18:23:37.986 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-04T18:23:38.154 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi019.front.sepia.ceph.com 2024-05-04T18:23:38.154 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-04T18:23:38.213 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi019.front.sepia.ceph.com 2024-05-04T18:23:38.213 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-04T18:23:38.271 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi019.front.sepia.ceph.com 2024-05-04T18:23:38.271 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-04T18:23:38.339 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-04T18:23:38.339 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-04T18:23:38.487 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-04T18:23:38.487 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-04T18:23:38.593 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-04T18:23:38.593 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-04T18:23:38.652 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi186.front.sepia.ceph.com 2024-05-04T18:23:38.653 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-04T18:23:38.709 INFO:tasks.ceph:Archiving mon data... 2024-05-04T18:23:38.710 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124/data/mon.a.tgz 2024-05-04T18:23:38.710 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-05-04T18:23:38.728 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.h0NFyp1DTT 2024-05-04T18:23:38.729 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.h0NFyp1DTT 2024-05-04T18:23:38.888 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.h0NFyp1DTT 2024-05-04T18:23:38.971 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.h0NFyp1DTT is 479KB 2024-05-04T18:23:39.041 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.h0NFyp1DTT 2024-05-04T18:23:39.058 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124/data/mon.c.tgz 2024-05-04T18:23:39.058 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-05-04T18:23:39.111 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.6fqdVPo08m 2024-05-04T18:23:39.111 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.6fqdVPo08m 2024-05-04T18:23:39.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-05-04T18:23:39.257 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.6fqdVPo08m 2024-05-04T18:23:39.339 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.6fqdVPo08m is 480KB 2024-05-04T18:23:39.437 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.6fqdVPo08m 2024-05-04T18:23:39.452 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124/data/mon.b.tgz 2024-05-04T18:23:39.452 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-05-04T18:23:39.470 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.POVl7l1Dc9 2024-05-04T18:23:39.470 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.POVl7l1Dc9 2024-05-04T18:23:39.625 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.POVl7l1Dc9 2024-05-04T18:23:39.719 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.POVl7l1Dc9 is 480KB 2024-05-04T18:23:39.791 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.POVl7l1Dc9 2024-05-04T18:23:39.805 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-04T18:23:39.806 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-04T18:23:39.808 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-04T18:23:40.175 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-05-04T18:23:40.381 INFO:tasks.ceph:Archiving crash dumps... 2024-05-04T18:23:40.382 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124/remote/smithi019/crash 2024-05-04T18:23:40.383 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-04T18:23:40.410 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124/remote/smithi186/crash 2024-05-04T18:23:40.411 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-04T18:23:40.433 INFO:tasks.ceph:Compressing logs... 2024-05-04T18:23:40.434 DEBUG:teuthology.orchestra.run.smithi019:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-04T18:23:40.454 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-04T18:23:40.477 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106640.log 2024-05-04T18:23:40.486 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39209.log 2024-05-04T18:23:40.486 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37904.log 2024-05-04T18:23:40.486 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106640.log: /var/log/ceph/ceph-client.admin.39209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39528.log 2024-05-04T18:23:40.486 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106640.log.gz 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106262.log 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106220.log 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39528.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37904.log.gz 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38449.log 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106010.log 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.106262.log.gz 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37188.log 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106010.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106220.log.gz 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39343.log 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36883.log 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.106010.log.gz 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37803.log 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37188.log.gz 2024-05-04T18:23:40.487 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.36883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36787.log 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38059.log 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36883.log.gz 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/ceph-client.admin.36787.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.106052.log 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37803.log.gz 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36787.log.gz 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38807.log 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38059.log: /var/log/ceph/ceph-client.admin.106052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39577.log 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38807.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38059.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84593.log 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106052.log.gz 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39577.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26953.log 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.39209.log.gz 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.84593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97347.log 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.tmp-client.admin.26953.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-04T18:23:40.488 INFO:teuthology.orchestra.run.smithi019.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26953.log.gz 2024-05-04T18:23:40.489 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.97347.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.84593.log.gz 2024-05-04T18:23:40.490 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37184.log 2024-05-04T18:23:40.490 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37946.log 2024-05-04T18:23:40.490 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37184.log: 91.6% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37184.log.gz 2024-05-04T18:23:40.490 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.97347.log.gz 2024-05-04T18:23:40.490 INFO:teuthology.orchestra.run.smithi019.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.39343.log.gz 2024-05-04T18:23:40.491 INFO:teuthology.orchestra.run.smithi019.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.38449.log.gz 2024-05-04T18:23:40.492 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37187.log 2024-05-04T18:23:40.492 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37946.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-04T18:23:40.492 INFO:teuthology.orchestra.run.smithi019.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.38807.log.gz 2024-05-04T18:23:40.492 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37189.log 2024-05-04T18:23:40.492 INFO:teuthology.orchestra.run.smithi019.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37946.log.gz 2024-05-04T18:23:40.495 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36967.log 2024-05-04T18:23:40.495 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37187.log.gz 2024-05-04T18:23:40.495 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39564.log 2024-05-04T18:23:40.496 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.36967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79584.log 2024-05-04T18:23:40.496 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37189.log.gz 2024-05-04T18:23:40.496 INFO:teuthology.orchestra.run.smithi019.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36967.log.gz 2024-05-04T18:23:40.496 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5/var/log/ceph/ceph-client.admin.39564.log: /var/log/ceph/ceph-client.admin.79584.log: --verbose -- /var/log/ceph/ceph-client.admin.38498.log 2024-05-04T18:23:40.496 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37883.log 2024-05-04T18:23:40.496 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38498.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-04T18:23:40.497 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37883.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37883.log.gz 2024-05-04T18:23:40.497 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35282.log 2024-05-04T18:23:40.499 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.audit.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.79584.log.gz 2024-05-04T18:23:40.499 INFO:teuthology.orchestra.run.smithi019.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.39564.log.gz 2024-05-04T18:23:40.499 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37823.log 2024-05-04T18:23:40.499 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.35282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37377.log 2024-05-04T18:23:40.499 INFO:teuthology.orchestra.run.smithi019.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35282.log.gz 2024-05-04T18:23:40.522 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32711.log 2024-05-04T18:23:40.522 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-04T18:23:40.522 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-04T18:23:40.522 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32711.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-04T18:23:40.522 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph.audit.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32711.log.gz 2024-05-04T18:23:40.522 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32791.log 2024-05-04T18:23:40.522 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-04T18:23:40.522 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32624.log 2024-05-04T18:23:40.522 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32915.log 2024-05-04T18:23:40.522 INFO:teuthology.orchestra.run.smithi186.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32791.log.gz 2024-05-04T18:23:40.522 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32624.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32915.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.4.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32624.log.gz 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr:gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32915.log.gz 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr: -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose/var/log/ceph/ceph-mgr.x.log: -- /var/log/ceph/valgrind/osd.6.log 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/valgrind/osd.6.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/valgrind/osd.7.log: /var/log/ceph/valgrind/osd.4.log: 64.0% 64.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32751.log 2024-05-04T18:23:40.523 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/valgrind/osd.5.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-05-04T18:23:40.524 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32871.log 2024-05-04T18:23:40.524 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32751.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32751.log.gz 2024-05-04T18:23:40.524 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32831.log 2024-05-04T18:23:40.524 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32871.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32871.log.gz 2024-05-04T18:23:40.524 INFO:teuthology.orchestra.run.smithi186.stderr: 92.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-04T18:23:40.524 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32671.log 2024-05-04T18:23:40.524 INFO:teuthology.orchestra.run.smithi186.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-04T18:23:40.524 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-04T18:23:40.524 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32831.log: /var/log/ceph/ceph-client.admin.32671.log: /var/log/ceph/ceph-mon.b.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32831.log.gz 2024-05-04T18:23:40.524 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32671.log.gz 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37823.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37823.log.gz 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38101.log 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38816.log 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38101.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37377.log.gz 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.39577.log.gz 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38101.log.gz 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.38498.log.gz 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39144.log 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106514.log 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76427.log 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104055.log 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39144.log.gz 2024-05-04T18:23:40.527 INFO:teuthology.orchestra.run.smithi019.stderr: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.76427.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.106514.log.gz 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32816.log 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.104055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37509.log 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.32816.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.76427.log.gz 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37509.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37509.log.gz 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.32816.log.gz 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106472.log 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33118.log 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106304.log 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.33118.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.106472.log.gz 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.104055.log.gz 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33118.log.gz 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38471.log 2024-05-04T18:23:40.528 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38568.log 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106430.log 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.106304.log.gz 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84592.log 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106430.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.106430.log.gz 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38866.log 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.84592.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.38816.log.gz 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr: 95.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38976.log 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38866.log: gzip 90.6% -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84592.log.gz 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38976.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.38568.log.gz 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.38976.log.gz 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.38866.log.gz 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-05-04T18:23:40.529 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104056.log 2024-05-04T18:23:40.530 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97348.log 2024-05-04T18:23:40.530 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.104056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106556.log 2024-05-04T18:23:40.530 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.97348.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.104056.log.gz 2024-05-04T18:23:40.530 INFO:teuthology.orchestra.run.smithi019.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.97348.log.gz 2024-05-04T18:23:40.530 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-04T18:23:40.530 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39495.log 2024-05-04T18:23:40.530 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106556.log: /var/log/ceph/ceph-osd.0.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106556.log.gz 2024-05-04T18:23:40.530 INFO:teuthology.orchestra.run.smithi019.stderr: 96.8% -- replaced with /var/log/ceph/ceph-client.admin.39528.log.gz 2024-05-04T18:23:40.530 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37009.log 2024-05-04T18:23:40.530 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37565.log 2024-05-04T18:23:40.531 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39495.log: /var/log/ceph/ceph-client.admin.37009.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37009.log.gz 2024-05-04T18:23:40.539 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106388.log 2024-05-04T18:23:40.540 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37565.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37565.log.gz 2024-05-04T18:23:40.546 INFO:teuthology.orchestra.run.smithi019.stderr: 94.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-04T18:23:40.552 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97346.log 2024-05-04T18:23:40.553 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37723.log 2024-05-04T18:23:40.553 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.97346.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.106388.log.gz 2024-05-04T18:23:40.556 INFO:teuthology.orchestra.run.smithi019.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.97346.log.gz 2024-05-04T18:23:40.565 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39485.log 2024-05-04T18:23:40.566 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39116.log 2024-05-04T18:23:40.566 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37723.log: /var/log/ceph/ceph-client.admin.39485.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.39495.log.gz 2024-05-04T18:23:40.567 INFO:teuthology.orchestra.run.smithi019.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37723.log.gz 2024-05-04T18:23:40.567 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-04T18:23:40.567 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97349.log 2024-05-04T18:23:40.575 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39116.log: /var/log/ceph/ceph-mon.c.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.38471.log.gz 2024-05-04T18:23:40.575 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-05-04T18:23:40.575 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.97349.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.39116.log.gz 2024-05-04T18:23:40.576 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-05-04T18:23:40.577 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/valgrind/osd.0.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-05-04T18:23:40.577 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-05-04T18:23:40.577 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/valgrind/mon.a.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-05-04T18:23:40.578 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-05-04T18:23:40.578 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/valgrind/osd.3.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-05-04T18:23:40.579 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-05-04T18:23:40.579 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/valgrind/osd.1.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-05-04T18:23:40.580 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-05-04T18:23:40.582 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/valgrind/mon.c.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-05-04T18:23:40.582 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70591.log 2024-05-04T18:23:40.582 INFO:teuthology.orchestra.run.smithi019.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.97349.log.gz 2024-05-04T18:23:40.582 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39351.log 2024-05-04T18:23:40.582 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.70591.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-05-04T18:23:40.582 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39036.log 2024-05-04T18:23:40.584 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39351.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.39351.log.gz 2024-05-04T18:23:40.585 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37743.log 2024-05-04T18:23:40.585 INFO:teuthology.orchestra.run.smithi019.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.70591.log.gz 2024-05-04T18:23:40.585 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70590.log 2024-05-04T18:23:40.587 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37743.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37743.log.gz 2024-05-04T18:23:40.587 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38959.log 2024-05-04T18:23:40.590 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.70590.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.70590.log.gz 2024-05-04T18:23:40.590 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37607.log 2024-05-04T18:23:40.594 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38959.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.39036.log.gz 2024-05-04T18:23:40.598 INFO:teuthology.orchestra.run.smithi019.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.39485.log.gz 2024-05-04T18:23:40.598 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34166.log 2024-05-04T18:23:40.599 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37607.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-04T18:23:40.599 INFO:teuthology.orchestra.run.smithi019.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.38959.log.gz 2024-05-04T18:23:40.601 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34166.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106682.log 2024-05-04T18:23:40.601 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37607.log.gz 2024-05-04T18:23:40.601 INFO:teuthology.orchestra.run.smithi019.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34166.log.gz 2024-05-04T18:23:40.601 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38895.log 2024-05-04T18:23:40.601 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38591.log 2024-05-04T18:23:40.601 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106682.log: /var/log/ceph/ceph-client.admin.38895.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106682.log.gz 2024-05-04T18:23:40.602 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37186.log 2024-05-04T18:23:40.610 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38591.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.38895.log.gz 2024-05-04T18:23:40.614 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39020.log 2024-05-04T18:23:40.616 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37186.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37186.log.gz 2024-05-04T18:23:40.616 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97350.log 2024-05-04T18:23:40.629 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106598.log 2024-05-04T18:23:40.645 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.97350.log: 91.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106094.log 2024-05-04T18:23:40.645 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.97350.log.gz 2024-05-04T18:23:40.645 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106598.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106598.log.gz 2024-05-04T18:23:40.651 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84594.log 2024-05-04T18:23:40.665 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106094.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.106094.log.gz 2024-05-04T18:23:40.670 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36443.log 2024-05-04T18:23:40.672 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.84594.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.84594.log.gz 2024-05-04T18:23:40.678 INFO:teuthology.orchestra.run.smithi019.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.39020.log.gz 2024-05-04T18:23:40.678 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38772.log 2024-05-04T18:23:40.679 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.36443.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36443.log.gz 2024-05-04T18:23:40.680 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39202.log 2024-05-04T18:23:40.694 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37499.log 2024-05-04T18:23:40.700 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39202.log: 92.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37429.log 2024-05-04T18:23:40.701 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37499.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.38772.log.gz 2024-05-04T18:23:40.702 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39202.log.gz 2024-05-04T18:23:40.703 INFO:teuthology.orchestra.run.smithi019.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37499.log.gz 2024-05-04T18:23:40.703 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36843.log 2024-05-04T18:23:40.704 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37429.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37429.log.gz 2024-05-04T18:23:40.704 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38464.log 2024-05-04T18:23:40.706 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.36843.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36843.log.gz 2024-05-04T18:23:40.706 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39203.log 2024-05-04T18:23:40.709 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38464.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.38464.log.gz 2024-05-04T18:23:40.709 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76425.log 2024-05-04T18:23:40.714 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39203.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.39203.log.gz 2024-05-04T18:23:40.714 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39618.log 2024-05-04T18:23:40.719 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.76425.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.76425.log.gz 2024-05-04T18:23:40.719 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105842.log 2024-05-04T18:23:40.723 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39618.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.39618.log.gz 2024-05-04T18:23:40.723 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105968.log 2024-05-04T18:23:40.725 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.105842.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105842.log.gz 2024-05-04T18:23:40.725 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106178.log 2024-05-04T18:23:40.726 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.105968.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105968.log.gz 2024-05-04T18:23:40.726 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39090.log 2024-05-04T18:23:40.728 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106178.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.106178.log.gz 2024-05-04T18:23:40.728 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39382.log 2024-05-04T18:23:40.732 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39090.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.39090.log.gz 2024-05-04T18:23:40.732 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79582.log 2024-05-04T18:23:40.747 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39382.log: 90.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106722.log 2024-05-04T18:23:40.753 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.79582.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.79582.log.gz 2024-05-04T18:23:40.753 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39382.log.gz 2024-05-04T18:23:40.753 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36925.log 2024-05-04T18:23:40.755 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106722.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.106722.log.gz 2024-05-04T18:23:40.755 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39350.log 2024-05-04T18:23:40.756 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.36925.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36925.log.gz 2024-05-04T18:23:40.756 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106136.log 2024-05-04T18:23:40.763 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39350.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.39350.log.gz 2024-05-04T18:23:40.763 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38017.log 2024-05-04T18:23:40.764 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106136.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.106136.log.gz 2024-05-04T18:23:40.765 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105884.log 2024-05-04T18:23:40.766 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38017.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38017.log.gz 2024-05-04T18:23:40.766 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105800.log 2024-05-04T18:23:40.768 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.105884.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105884.log.gz 2024-05-04T18:23:40.768 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39593.log 2024-05-04T18:23:40.769 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.105800.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105800.log.gz 2024-05-04T18:23:40.769 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105926.log 2024-05-04T18:23:40.783 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39593.log: 90.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79583.log 2024-05-04T18:23:40.785 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.105926.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105926.log.gz 2024-05-04T18:23:40.785 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39593.log.gz 2024-05-04T18:23:40.785 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38795.log 2024-05-04T18:23:40.787 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.79583.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.79583.log.gz 2024-05-04T18:23:40.787 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39612.log 2024-05-04T18:23:40.794 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38795.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.38795.log.gz 2024-05-04T18:23:40.794 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38618.log 2024-05-04T18:23:40.808 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36656.log 2024-05-04T18:23:40.814 INFO:teuthology.orchestra.run.smithi186.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-04T18:23:40.814 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39336.log 2024-05-04T18:23:40.821 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.36656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37183.log 2024-05-04T18:23:40.822 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39336.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36656.log.gz 2024-05-04T18:23:40.827 INFO:teuthology.orchestra.run.smithi019.stderr: 89.8% 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38618.log.gz 2024-05-04T18:23:40.835 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39336.log.gz 2024-05-04T18:23:40.841 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38691.log 2024-05-04T18:23:40.841 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37183.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37183.log.gz 2024-05-04T18:23:40.854 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33145.log 2024-05-04T18:23:40.867 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70589.log 2024-05-04T18:23:40.867 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.33145.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33145.log.gz 2024-05-04T18:23:40.871 INFO:teuthology.orchestra.run.smithi019.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.39612.log.gz 2024-05-04T18:23:40.880 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37441.log 2024-05-04T18:23:40.883 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.70589.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.70589.log.gz 2024-05-04T18:23:40.893 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39287.log 2024-05-04T18:23:40.893 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37441.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37441.log.gz 2024-05-04T18:23:40.903 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36739.log 2024-05-04T18:23:40.909 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39287.log: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.39287.log.gz 2024-05-04T18:23:40.910 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39210.log 2024-05-04T18:23:40.910 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.36739.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36739.log.gz 2024-05-04T18:23:40.911 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38841.log 2024-05-04T18:23:40.914 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39210.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.39210.log.gz 2024-05-04T18:23:40.914 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38686.log 2024-05-04T18:23:40.925 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38841.log: 92.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37190.log 2024-05-04T18:23:40.931 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38434.log 2024-05-04T18:23:40.931 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38841.log.gz 2024-05-04T18:23:40.944 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39446.log 2024-05-04T18:23:40.945 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38434.log: /var/log/ceph/ceph-client.admin.37190.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37190.log.gz 2024-05-04T18:23:40.954 INFO:teuthology.orchestra.run.smithi019.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.38434.log.gz 2024-05-04T18:23:40.955 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106346.log 2024-05-04T18:23:40.968 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104057.log 2024-05-04T18:23:40.968 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.106346.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.106346.log.gz 2024-05-04T18:23:40.976 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33299.log 2024-05-04T18:23:40.977 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.104057.log: 94.1% -- replaced with /var/log/ceph/ceph-client.admin.39446.log.gz 2024-05-04T18:23:40.988 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37191.log 2024-05-04T18:23:40.992 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.33299.log: 88.1% 91.0% -- replaced with /var/log/ceph/ceph-client.admin.104057.log.gz 2024-05-04T18:23:41.001 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38416.log 2024-05-04T18:23:41.002 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37191.log: -- replaced with /var/log/ceph/ceph-client.admin.33299.log.gz 2024-05-04T18:23:41.002 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37191.log.gz 2024-05-04T18:23:41.027 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37433.log 2024-05-04T18:23:41.035 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38394.log 2024-05-04T18:23:41.049 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.37433.log: /var/log/ceph/ceph-client.admin.38394.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37433.log.gz 2024-05-04T18:23:41.086 INFO:teuthology.orchestra.run.smithi019.stderr: 94.4% -- replaced with /var/log/ceph/ceph-client.admin.38686.log.gz 2024-05-04T18:23:41.108 INFO:teuthology.orchestra.run.smithi019.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.38394.log.gz 2024-05-04T18:23:41.112 INFO:teuthology.orchestra.run.smithi019.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.38416.log.gz 2024-05-04T18:23:41.454 INFO:teuthology.orchestra.run.smithi019.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.38691.log.gz 2024-05-04T18:23:42.020 INFO:teuthology.orchestra.run.smithi019.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.38591.log.gz 2024-05-04T18:23:42.281 INFO:teuthology.orchestra.run.smithi186.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-04T18:23:42.731 INFO:teuthology.orchestra.run.smithi019.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-04T18:23:44.717 INFO:teuthology.orchestra.run.smithi019.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-04T18:23:47.562 INFO:teuthology.orchestra.run.smithi186.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-04T18:23:47.837 INFO:teuthology.orchestra.run.smithi186.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-04T18:23:47.961 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-04T18:23:48.107 INFO:teuthology.orchestra.run.smithi186.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-04T18:23:48.522 INFO:teuthology.orchestra.run.smithi186.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-04T18:23:48.524 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-04T18:23:48.524 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m8.035s 2024-05-04T18:23:48.524 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m31.096s 2024-05-04T18:23:48.524 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m1.012s 2024-05-04T18:23:48.629 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-04T18:23:48.742 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-04T18:23:49.599 INFO:teuthology.orchestra.run.smithi019.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-04T18:23:49.602 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-05-04T18:23:49.602 INFO:teuthology.orchestra.run.smithi019.stderr:real 0m9.134s 2024-05-04T18:23:49.602 INFO:teuthology.orchestra.run.smithi019.stderr:user 0m40.471s 2024-05-04T18:23:49.602 INFO:teuthology.orchestra.run.smithi019.stderr:sys 0m1.364s 2024-05-04T18:23:49.602 INFO:tasks.ceph:Archiving logs... 2024-05-04T18:23:49.602 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124/remote/smithi019/log 2024-05-04T18:23:49.604 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-04T18:23:54.987 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124/remote/smithi186/log 2024-05-04T18:23:54.994 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-04T18:23:59.459 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-05-04T18:23:59.460 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-04T18:23:59.470 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_f273489c6dcd4bc88409993babd09dd99491162c/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-05-04T18:23:59.470 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-04T18:23:59.470 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-04T18:23:59.473 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-04T18:23:59.558 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-04T18:23:59.567 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-04T18:23:59.567 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-04T18:23:59.569 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-04T18:23:59.583 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-05-04T18:23:59.586 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-05-04T18:23:59.588 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-04T18:23:59.588 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-05-04T18:23:59.588 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv01.front.sepia.ceph.com 3 6 377 30 +806us[ +802us] +/- 45ms 2024-05-04T18:23:59.588 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 -1037us[-1037us] +/- 68ms 2024-05-04T18:23:59.588 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -170us[ -170us] +/- 66ms 2024-05-04T18:23:59.588 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T18:23:59.592 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-04T18:23:59.592 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-05-04T18:23:59.592 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 +808us[ +813us] +/- 45ms 2024-05-04T18:23:59.592 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 -1029us[-1024us] +/- 68ms 2024-05-04T18:23:59.592 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 -164us[ -159us] +/- 66ms 2024-05-04T18:23:59.592 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-04T18:23:59.593 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-04T18:23:59.601 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-04T18:23:59.602 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-04T18:23:59.646 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-04T18:23:59.713 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-04T18:23:59.776 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-04T18:23:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T18:23:59.805 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-04T18:23:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T18:23:59.833 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-04T18:23:59.842 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-04T18:23:59.863 INFO:teuthology.task.internal:Duration was 3246.965067 seconds 2024-05-04T18:23:59.864 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-04T18:23:59.913 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-04T18:23:59.914 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-04T18:23:59.918 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-04T18:23:59.954 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-04T18:23:59.956 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-04T18:24:00.291 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-04T18:24:00.292 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2024-05-04T18:24:00.292 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-04T18:24:00.354 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-05-04T18:24:00.354 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-04T18:24:00.387 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-04T18:24:00.388 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-04T18:24:00.397 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-04T18:24:00.455 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-04T18:24:00.455 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-04T18:24:00.464 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-04T18:24:00.595 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-04T18:24:00.604 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-04T18:24:00.605 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-04T18:24:00.629 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-04T18:24:00.652 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-04T18:24:00.661 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-04T18:24:00.671 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-04T18:24:00.691 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2024-05-04T18:24:00.714 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-05-04T18:24:00.734 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-04T18:24:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T18:24:00.759 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-04T18:24:00.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-04T18:24:00.788 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-04T18:24:00.798 INFO:teuthology.task.internal:Transferring archived files... 2024-05-04T18:24:00.798 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124/remote/smithi019 2024-05-04T18:24:00.798 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-04T18:24:00.854 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124/remote/smithi186 2024-05-04T18:24:00.854 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-04T18:24:00.885 INFO:teuthology.task.internal:Removing archive directory... 2024-05-04T18:24:00.885 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-04T18:24:00.887 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-04T18:24:00.934 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-04T18:24:00.947 INFO:teuthology.task.internal:Not uploading archives. 2024-05-04T18:24:00.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-04T18:24:00.972 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-04T18:24:00.973 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-04T18:24:00.975 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-04T18:24:00.988 INFO:teuthology.orchestra.run.smithi019.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 4 18:24 /home/ubuntu/cephtest 2024-05-04T18:24:00.989 INFO:teuthology.orchestra.run.smithi186.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 4 18:24 /home/ubuntu/cephtest 2024-05-04T18:24:00.991 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-04T18:24:01.022 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-04T18:24:01.074 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-05-04T18:24:01.074 INFO:teuthology.nuke:Checking targets against current locks 2024-05-04T18:24:01.099 DEBUG:teuthology.nuke:shortname: smithi019 2024-05-04T18:24:01.099 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-04T18:24:01.124 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-04 17:24:11.240129', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOvty0GMgfRqiueBK+9zrw9w+jUpACO7qhUu4rXcOfp2gxDXptEzxgA9fTtDwyweBBm7WSF980wwDx3iJ6RO9Bw='} 2024-05-04T18:24:01.127 DEBUG:teuthology.nuke:shortname: smithi186 2024-05-04T18:24:01.127 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-04T18:24:01.151 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687124', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-04 17:24:11.238694', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHYzvai/rL2inX2fbEzwz6S4DYA4Fbh2e/rKxOZaVRoBX4sR4J8ehcnFxUBq0G36PrKUFqP287tjivJTsIZ7S2w='} 2024-05-04T18:24:01.183 INFO:teuthology.orchestra.console.smithi019:Power off 2024-05-04T18:24:01.183 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-04T18:24:01.219 INFO:teuthology.orchestra.console.smithi186:Power off 2024-05-04T18:24:01.220 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-04T18:24:01.318 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-05-04T18:24:01.319 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T18:24:01.329 DEBUG:teuthology.orchestra.console.smithi019:power off output: Chassis Power Control: Down/Off 2024-05-04T18:24:01.329 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T18:24:01.343 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-05-04T18:24:01.357 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-05-04T18:24:05.344 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T18:24:05.359 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T18:24:05.457 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-05-04T18:24:05.468 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-05-04T18:24:09.459 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T18:24:09.469 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-04T18:24:09.570 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-05-04T18:24:09.579 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is off 2024-05-04T18:24:09.672 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-05-04T18:24:09.680 INFO:teuthology.orchestra.console.smithi019:Power off completed 2024-05-04T18:24:09.785 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} duration: 3246.96506690979 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-05-04T18:24:09.785 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-04T18:24:09.866 INFO:teuthology.run:FAIL